denise | (no subject)

I've seen people wondering why imports are taking so @)#$(*& long. This might help shed some light:

#229690254 2011-04-08 08:13:11 [38 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

#229690299 2011-04-08 08:13:31 [38 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

#229691506 2011-04-08 08:17:12 [34 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Entries
State: failed at least once, will retry very soon, priority undefined.

#229693287 2011-04-08 08:22:58 [29 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

#229693552 2011-04-08 08:23:24 [28 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

#229694558 2011-04-08 08:26:19 [25 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Entries
State: failed at least once, will retry very soon, priority undefined.

#229696494 2011-04-08 08:30:26 [21 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Entries
State: failed at least once, will retry very soon, priority undefined.

#229697174 2011-04-08 08:31:00 [21 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Entries
State: failed at least once, will retry very soon, priority undefined.

#229699435 2011-04-08 08:37:17 [14 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Entries
State: failed at least once, will retry very soon, priority undefined.

#229699747 2011-04-08 08:38:02 [14 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Entries
State: failed at least once, will retry very soon, priority undefined.

#229700502 2011-04-08 08:39:28 [12 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

#229701320 2011-04-08 08:41:23 [10 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Verify
State: failed at least once, will retry in Never, priority undefined.

#229704262 2011-04-08 08:43:44 [8 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

#229705151 2011-04-08 08:44:30 [7 minutes ago] in DW::Worker::ContentImporter::LiveJournal::Comments
State: failed at least once, will retry very soon, priority undefined.

1) LJ is still under wicked load due to the DDoS they're experiencing. This can mean the site timing out when we try to contact them for the data.

2) Every time LJ is inaccessable when the importer tries to hit it, the job goes on pause to retry again in a few seconds. It retries up to five times before giving up.

3) There are lots and lots and lots of people trying to import right now. (This means that imports from sites other than LJ will be slow too, because they enter the queue behind all the LJ import jobs.)

If I had to guess, I'd say the existing import queue as it stands right this very minute would take at least 24 hours to clear, because of a combination of the three above items. Please, please, please be patient!

Flat | Top-Level Comments Only

Passing the word along as best I can, and I saw numerous tweets to that effect too from other friends.

Thank you, again, for trying to keep us in the loop, and I think a lot of us will try and keep others there!

Is it reasonable to say that right now is not the best time to attempt to import from LJ?

I understand that people want to protect their content, but with their servers under attack, importing right now is just adding to the strain, and clearly their servers cannot keep up.

It is totally reasonable, yeah :) I included a bit about that in this week's

dw_news, but I think the message got buried. I also added it to our Twitter status feed and the Support board, edited the text of the importer page asking people to hold off for a bit, and posted just now in

dw_maintenance.

I totally grok the "ack must back up!" impulse! but lord are things overwhelmed right now. One of their ops guys said that traffic is something like 10 times normal, and they can only handle about 30% of it.

Gah. I've been vocal in my disapproval of some things LJ has done, but this is unconscionable.

This reminds me that you mentioned you were willing to teach programming to help out Dreamwidth. I'm still interested in helping out with small projects. Perhaps you have a few links of interest that I can research with? I even braved the suggestion queue a few minutes ago to put a few more ideas forth. I also wouldn't mind being teamed up with a programmer to write the html/css end of their projects, too.

I think the best way to start with that is to drop by #dw_dev in IRC or make a post to

dw_dev_training talking about what you want to do.

That's #dreamwidth-dev, I think :)

Edited 2011-04-09 01:26 (UTC)

Ack, you're right! Fortunately the IRC wiki page will have the right stuff.

Suggestion I just thought of: maybe put an entry like this in

dw_maintenance? Or would that be a bad idea since it's not really maintenance on our part...

yeah, I stuck it in Twitter, the BBB, and on the import page itself, but clearly people are looking for more information, so I just did a dw-maint post too!

Woo woo!

I figured this sort of thing would be happening. I'm to the point where I'm not even retrying to crosspost from DW to LJ. If it fails the first time, I give it a good chunk of time before retrying. And I'm waiting for everything to be much more stable on the LJ side before helping my other half with doing an import/backup of her LJ again.

Yep.

Hell, LJ's been flagging ME, a meatbag, as a bot. Not surprising that it's flagging the importer as a bot.

Thanks a lot. I've imported all my posts about my kids, it was very important for me

This is what I get for not getting around to it (importing stuff over to DW). I hope things are going okay for DW generally. This must be a busy time.

Very busy, yes! Exciting. But busy. :)

Is it worth having multiple queues, so that non-LJ imports are less affected?

Or, if an import from site X fails, switching to the next import from a different site, before going back to site X again?

Until there's more of a diverse base of sites that people can import from, it's not worth the time it would take to code it. Something like 98% of our imports are from LJ right now.

(We're very close to having other import sources/protocols working, but mapping, say, Wordpress's entries to DW's entries is kind of hard.)

Good point - that does make sense.

I just wanted to say thank you for this feature; I know this is #oldpost and #threadzombieism here, but I wanted to thank you for the importer... i've had a DW journal for about a year and a half, but haven't done much with it. The recent issues at LJ had me worried for the loss of irreplaceable data... and with facebook stealing the majority of my active friends there away, I figured what the hey... I have little to lose.

And now, I've found that DW is my default. I LIKE it here better. It's cleaner, more stable, and truthfully... it's the ethics here that have really drawn me in. It feels like Old LJ.

I still love LJ, and I'll have an account there as long as they let me... but I think I'll make this my new home base. :)

I'm glad you like it here :)

If you guys keep up the exemplary company philosophy and work ethic... I think you'll set a whole new standard for web 2.0 business.

I can only hope so! It would be nice to have a mini-revolution in the Web 2.0 world, heh.

Is it about that time again? It's Tuesday! We got hit hard with a load of spam late last night, but it appears to have been relatively unsophisticated. All deletion of spam that is spam is always appreciated etc.