Import Mastodon (Beta)

Resolved TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago
I know it’s still beta, but there’s zero documentation (that I can see) for the Import Mastodon feature. (Is this even the right forum to be asking this?) All I can see under Tools>Import>Mastodon(Beta) is “Run importer”, and I don’t want to click that in case it immediately starts importing all the posts from the Mastodon account I’m logged into in my browser.

I don’t really think that’s what would happen, but with no “Details”, I can’t be sure. What I’d like to know is:
- Do we get to select a file/archive to use as the import source?
- If so, which type of file? Hopefully a previously downloaded .zip Mastodon archive.
- Does every imported Masto post become a single WP post, with any attached media being sent to the WP Media library, and embedded into the post?
- Does each created WP post have the same publish date as the original Masto post? I hope so. (I don’t want 500 “new” posts at the top of my WP feed)
- And, while I’d want each post to be “available” via my WP’s APub , I wouldn’t want to push out 500 “new” old posts to my blog’s followers. How would that work?
- What is the title of each new WP post? Would it be the first sentence or few words of the post, or something dull but sensible like the original post ID number, eg 123456789012345 ? (I can see that the latter might be useful, so you could easily associate, say, myoldinstance.social/@myusername/123456789012345 with
  mywordpresssite.com/blog/123456789012345 )
- Do Favourite/Boost counts get imported, and/or shown? And who they were by?
- If it all goes horribly wrong, are there any Undo options?
Sorry for all the questions! I’m interested in trying it out, but I want to know exactly what I’m trying.

Viewing 15 replies - 1 through 15 (of 37 total)

1 2 3 →

Plugin Contributor Konstantin Obenland
(@obenland)

1 year, 1 month ago

Hi there! You’re in the right place 🙂

There is a bit more information once you clicked on “Run Importer”—although I understand how that wording can be confusing. It really just loads the importer, which then takes you through a couple of setup steps before it asks you to click an “Import” button that starts the actual import.

Do we get to select a file/archive to use as the import source?

Yes, absolutely.

If so, which type of file? Hopefully a previously downloaded .zip Mastodon archive.

That is correct!

Does every imported Masto post become a single WP post, with any attached media being sent to the WP Media library, and embedded into the post?

Yes! It’s pretty basic at this point, it just adds any images it finds to the end of the post. Part of where the “Beta” comes in is that there’s not much more support for other media types at this point.

Does each created WP post have the same publish date as the original Masto post? I hope so. (I don’t want 500 “new” posts at the top of my WP feed)

It does!

And, while I’d want each post to be “available” via my WP’s APub , I wouldn’t want to push out 500 “new” old posts to my blog’s followers. How would that work?

Correct. The plugin skips putting posts into the Outbox when WordPress is in “Import Mode”. I suppose we could just suspend the sending out part and still put them in the Outbox, but for now that was the simplest way to achieve that.

What is the title of each new WP post? Would it be the first sentence or few words of the post, or something dull but sensible like the original post ID number, eg 123456789012345 ? (I can see that the latter might be useful, so you could easily associate, say, myoldinstance.social/@myusername/123456789012345 with
mywordpresssite.com/blog/123456789012345 )

The title will be empty. A lot of sites have lists of posts and a list of IDs is probably not the most helpful for most people. The Importer assigns each post the “Status” post format, so that themes that support it can display them as such. They usually don’t have a title.

Do Favourite/Boost counts get imported, and/or shown? And who they were by?

Not at this point. There’s currently no built-in mechanism to represent these in WordPress, and to do it correctly the Importer would probably have to make two remote requests per post to get the likes and reposts list, and then a request for each individual actor to get all the information needed to save them as “comments” and populate the Reactions block. It just wouldn’t scale well.

Very much open to ideas and suggestions on how to improve that, though!

If it all goes horribly wrong, are there any Undo options?

I’m afraid not 🙂

The imported posts can be deleted with the usual bulk actions in the posts list (the amount of displayed items can be adjusted in screen options). If you’re comfortable with coding a small script—each post has a _source_id meta with the original post id, that could be used in a SQL query.

—

If you’re looking for a low risk way to trying it out, you could set up a test account on a Mastodon instance somewhere, create a single post, and generate a small export from that.

Other than that, I hope this helps! Let me know what you think of the experience when you had a chance to test it!

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

That’s all really helpful info. 🙂 I’ll report back when I get around to trying it. Good idea about a test account. But the account I want to import has quite a few video posts, so I guess I’ll have to wait a while before I can import that (fully, at least).
Thanks very much!

Plugin Contributor Konstantin Obenland
(@obenland)

1 year, 1 month ago

We’ll have some basic video and audio file support in the next version of the plugin.

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

Reporting back on a test import:
I created a new Masto account on a new server, and made four posts:
1. First post, just some text.
2. Another, text plus hash tags. #offgridpod #volleyclub
3. Third, text, with a pic with alt text and a hash tag. #offgridpod
4. Fourth, text, with a short mp4 video file(x264/aac), with alt text, and a hash tag. #kendama

Downloaded the Masto archive of the 4 posts, and ran the import. All nice and easy, and 1, 2 & 3 all went fine, creating title-less “status” posts. #4 Created a post, but didn’t have the video file attached – as expected with the beta description.

Ah, just noticed that the image was imported into my WP Media library (as expected), and the mp4 file has been too! So it was imported, but just not embedded into the post. I guess this is something you already know.

The alt text from the video has been imported into the WP Media file’s Title and Description fields. The alt text from the image has been imported into the WP Media file’s Title and Description fields, but *not* into the WP Media file’s Alt Text field. However, looking at Page Source, I can see that the image embedded into the post *does* have the Alt Text.

The statuses were tagged with WP-native tags that match the ones in the content (good!), but the hash tags in the body of the post link to a tag search on the original Masto host. I guess that makes sense, since it’s just replicating the original Masto posts’ href links around the tags. Although in my case, the archive I actually want to back up is from a dead server, so those will be dead links. Okay, I guess that’s my problem, not yours. 🙂

I had a minor panic that the date stamps of the posts were wrong, as they all appeared at the top of my WP feed. Then I realised that’s because they were brand new Masto posts, all newer than any other posts on my WP, so that was correct. My WP server is on UK (current summer) time, but the post time stamps were an hour behind (the Masto server must be on UTC), so that confirms that the imported time stamps *do* match the times of original toot creation. Cool!

The post urls are of the form [baseurl]/1234-3 , which isn’t especially helpful, but no less so than Mastodon status urls of 123456789012345 , I suppose. *shrug*

My next hoop to jump through, after there’s video support, will be the Import function’s file size limit of 512MB, as my archive file is just over, at 540MB. I reckon I’ll probably be ably to do something similar to what I did via Posty to get around that.

So, all looking good, thanks!

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

Oh, and all are posted under the WP category of “Uncategorised”. Could that be something more helpful, like “Mastodon”, “toot”, “status”, “import”, “imported post” or something?

Plugin Contributor Konstantin Obenland
(@obenland)

1 year, 1 month ago

5.8.0 was just release, which includes video and audio support for the Mastodon importer 🙂

Thanks for trying it out and sharing your feedback!

The alt text from the image has been imported into the WP Media file’s Title and Description fields, but *not* into the WP Media file’s Alt Text field.

Yeah, I noticed that, too! That should be fixed in the latest version now.

in my case, the archive I actually want to back up is from a dead server, so those will be dead links.

There should be plugins that make it easy(or at least easier) to bulk-replace URLs in post content, that should help with that.

Could [categories] be something more helpful, like “Mastodon”, “toot”, “status”, “import”, “imported post” or something?

At some point we could maybe add a category selector in the first step? I’m sure there are folks who feel as strongly that there shouldn’t be a category at all 🙂

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

Ha, my timing is terrible! 🙂 I’ll try again with 5.8, thanks.
Noted re a bulk-replacer, yep. And yes, I did mean optional re-categorising. That would be cool – and maybe adding tags too? Although perhaps both is overkill.

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

Tried again with 5.8:
Deleted the 4 posts I created last time, then deleted the two files from the Media Library. Re-imported the same .zip archive, and got 4 new posts, this time with the video embedded in the post, and the image’s Alt Text in its Alt Text field. 🙂 Yay, thanks!

One really small thing: There are usually two line breaks between the date stamp of the WP post and its content (in my theme, at least), but on these imported posts, there appears to be three. Looking at View Source, there are no extra <br>s, so I think it might just be that there’s a line of a couple of spaces (tabs?) in the code after <div class="entry-content">. As it is, the date stamp looks half-way between one post and the next. See… https://social.vivaldi.net/@TheVoidTLMB/114394053143071337
Cheers, Void

Plugin Contributor Konstantin Obenland
(@obenland)

1 year, 1 month ago

I’m glad it works! Mostly 🙂

Could you paste the post contents from the Code view in the Editor? And/or link to the post where that happens?

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

Hi, here’s the link to the first of the 4 posts. The apparent extra line break appears on all four of them.
https://tlmb.net/blog/3746-2/
Compare with the post below it “5 Years…” or any older than that.
(I can’t paste the code here, it just appears as the post text, even though I’m using the inline code button.)

Plugin Contributor Konstantin Obenland
(@obenland)

1 year, 1 month ago

It looks like that gap is a result of a missing heading. The theme doesn’t seem to account for posts without them.

Thread Starter TheVoidTLMB
(@thevoidtlmb)

1 year, 1 month ago

Okay, good to know it’s the theme, not the importer, at fault. Thanks.

Thread Starter TheVoidTLMB
(@thevoidtlmb)

9 months, 4 weeks ago

“At some point we could maybe add a category selector in the first step? I’m sure there are folks who feel as strongly that there shouldn’t be a category at all :-)” – Konstantin
“…yes, I did mean optional re-categorising. That would be cool – and maybe adding tags too? Although perhaps both is overkill.” – Void

@obenland I know you’re busy with other things, but a small bump for this request. I have 3 Mastodon archives I’m waiting to import, and it’d be really nice to add multiple Categories of (say) OldInstanceName, Mastodon and Microblog to each of them, to help me search for old posts a little more easily. (Having thought about it, now I don’t think I’d want to add tags too.)

Plugin Contributor Konstantin Obenland
(@obenland)

9 months, 1 week ago

Yeah, I’m afraid it’s not top of the list right now
Thread Starter TheVoidTLMB
(@thevoidtlmb)

7 months ago
@obenland Update: I decided to have another try at this.
First I installed this plugin to my site:
https://en-gb.ww.wp.xz.cn/plugins/wpapps-show-post-id/ (to help me find the imported posts once they were there, and change the Category that they were filed in). No problem with that.

I have 4 different Mastodon archives:
#1 A small one from Mastodon.online (account now deleted)
#2 A larger one (71MB) from qoto.org (account redirected to… #3)
#3 One a little larger than 512MB, from moth.social (instance now closed)
#4 A very small test archive (only 4 posts) from mastodon.social

#s1&2 are .tar.gz files, not zip, so on trying #1 I got the error message ~”that’s not a zip file”. Fair enough! I uncompressed the .tar.gz, and got a folder on my hard drive with the expected contents. I then zipped this folder, so that I had a .zip file to upload to the importer. Then I ran the importer again and went through the steps of

Choose File (the .zip file of #1 I’d just made on my Mac)
Upload file & import
Assign author, tick “Download & import file attachments” > Import

And I got the error message:
“Sorry, there has been an error.
The archive does not contain an Outbox file, please try again.”

The folder does contain a file called outbox.json

I then started again, going through the whole process, this time using archive #2, and again using a zipped file of the folder extracted from the .tar.gz. Same result.

Then I tried re-zipping everything inside the folder (but not the folder itself), and using that zip. This time I got:

There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.

The email:
```
Hii!

WordPress has a built-in feature that detects when a plugin or theme causes a fatal error on your site, and notifies you with this automated email.

In this case, WordPress caught an error with one of your plugins, ActivityPub.

First, visit your website (https://tlmb.net/blog/) and check for any visible issues. Next, visit the page where the error was caught (https://tlmb.net/blog/blog/wp-admin/admin.php?import=mastodon&step=2) and check for any visible issues.

Please contact your host for assistance with investigating this issue further.

If your site appears broken and you can't access your dashboard normally, WordPress now has a special "recovery mode". This lets you safely login to your dashboard and investigate further.

[ link removed ]

To keep your site safe, this link will expire in 1 day. Don't worry about that, though: a new link will be emailed to you if the error occurs again after it expires.

When seeking help with this issue, you may be asked for some of the following information:
WordPress version 6.8.3
Active theme: Sundance (version 1.1.3)
Current plugin: ActivityPub (version 7.5.0)
PHP version 8.1.31

Error Details
=============
An error of type E_ERROR was caused in line 553 of the file /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/functions.php. Error message: Uncaught Error: Cannot use object of type stdClass as array in /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/functions.php:553
Stack trace:
#0 /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/functions.php(534): Activitypub\extract_recipients_from_activity_property('bto', Array)
#1 /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/functions.php(604): Activitypub\extract_recipients_from_activity(Array)
#2 /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/wp-admin/import/class-mastodon.php(286): Activitypub\is_activity_public(Array)
#3 /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/wp-admin/import/class-mastodon.php(246): Activitypub\WP_Admin\Import\Mastodon::import_posts()
#4 /home/tlmb/tlmb.net/blog/wp-content/plugins/activitypub/includes/wp-admin/import/class-mastodon.php(81): Activitypub\WP_Admin\Import\Mastodon::import()
#5 /home/tlmb/tlmb.net/blog/wp-admin/admin.php(365): Activitypub\WP_Admin\Import\Mastodon::dispatch()
#6 {main}
 thrown
```
The site didn’t appear to be broken in any way, just that the import had failed. The next day (today) I tried again with archive #4 – a more recently downloaded file – in case it was something to do with a change in how Mastodon archives are formatted. I used the original downloaded .zip file, which includes the folder and contents. I got the same “There has been a critical error on this website.” error message. This time though, I did not receive an email about the error (perhaps because it was less than 24 hours since the last error?). Again, the site seems otherwise unaffected.

Archive #4 is the same one I previously successfully tested the Import with (see earlier in this thread).

Any ideas? Does this look like an AP issue, or should I contact my host?
Thanks for looking.

Viewing 15 replies - 1 through 15 (of 37 total)

1 2 3 →

You must be logged in to reply to this topic.

Tags