View previous topic :: View next topic |
Author |
Message |
Haggis
Joined: 14 Sep 2003 Posts: 27 Location: Brisbane |
Posted: Wed Oct 01, 2003 5:25 pm Post subject: Historical data |
|
|
There is a lot of interesting and useful information that is still at the old Yahoo JP1 group. Is there an easy way to migrate useful posts to this forum?
Haggis
|
|
Back to top |
|
|
Mark Pierson Expert
Joined: 03 Aug 2003 Posts: 3017 Location: Connecticut, USA |
Posted: Wed Oct 01, 2003 6:42 pm Post subject: |
|
|
Yahoo doesn't make it easy to do anything.
There's no way that I've found (at the admin level) to get any message content out of the Yahoo group so that it could be rolled in over here. That's why Rob has said the old group will remain intact as long as necessary to serve as an archive. While obviously not the best solution, it beats losing all that history.
Of course, if you're feeling ambitious, you could open EVERY message over there and copy & paste it into a new one over here! _________________ Mark |
|
Back to top |
|
|
The Robman Site Owner
Joined: 01 Aug 2003 Posts: 21271 Location: Chicago, IL |
Posted: Wed Oct 01, 2003 7:35 pm Post subject: |
|
|
Hey there Haggis, I thank you for stepping forward and offering your services. Please feel free to go through the Yahoo archives and identify all the posts that you think are useful and list their MSG# here and someone will copy them.
I am joking of course, unless of course you're just crazy enough to try actually doing that. In which case, please report back towards the end of next year and let us know your progress! _________________ Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help! |
|
Back to top |
|
|
Dabbith
Joined: 04 Aug 2003 Posts: 55 Location: Anonia, CT |
Posted: Mon Oct 06, 2003 9:45 am Post subject: |
|
|
Rob,
Thinking about the historical data...
I've actually got the messages from 6/6/2002 on archived into an outlook data file. I'm sure there are others that have similar archives. I don't think it would be too much trouble to spend a bit of time writing a macro to try and post these messages, but you'd need to set up a special section for them. I'd also need to know if anyone had the earlier posts.
Another option that might be worth investigating would be for me to write a macro that attempted to harvest the data from Yahoo and then add it to this site, that way we could have all of the posts here.
One of the problems would be that all of the posts would appear to come from me. I would at least include the original poster and date posted in the body.
Is there any way to add posts in a bulk format?
If you're interested in this idea, I'll see if I can come up with a way to scrape the data from Yahoo.
Dan |
|
Back to top |
|
|
The Robman Site Owner
Joined: 01 Aug 2003 Posts: 21271 Location: Chicago, IL |
Posted: Mon Oct 06, 2003 10:12 am Post subject: |
|
|
Hey Dan,
Insteresting idea. Would your macros be able to join posts together is thread format?
As for what's possible with these phpbb forums, I'm not entirely sure. If you want to do some research, go to the http://www.phpbb.com web site and take a look.
What might be easier would be to have your macros create regular HTML files with the correct user ids, etc, then I could just load these up as regular web site pages. _________________ Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help! |
|
Back to top |
|
|
Dabbith
Joined: 04 Aug 2003 Posts: 55 Location: Anonia, CT |
Posted: Wed Oct 08, 2003 9:16 am Post subject: |
|
|
I started to look into it a bit. Apperently, there are converters from several other forums to phpBB. Unfortunatly, Yahoo is not one of them. Also unfortunate is that all of them are written in PHP which I don't really have time to learn right now. There was some talk of trying to create a converter for Yahoo in March, but it seems to have died off. I think I'm going to try and scrape the data from yahoo into a database. If I manage that, I'll re-visit getting the data to this forum.
Is there anyone around who knows php?
If we could find someone, they might be able to create/modify a converter to import the database data. |
|
Back to top |
|
|
aberguerand Advanced Member
Joined: 11 Aug 2003 Posts: 257 Location: Lausanne, VD, Switzerland |
Posted: Wed Oct 08, 2003 10:34 am Post subject: |
|
|
There is a tool (Perl script) that can scrap the contents of a Yahoo Group and save it as an Unix email archive :
http://www.lpthe.jussieu.fr/~zeitlin/yahoo2mbox.html
There seems however that Yahoo places a limit on the quantity of data one user can download for a given period of time, so the whole archive might need to be get in several passes, and/or in a coordinated effort between several users. |
|
Back to top |
|
|
JCTerrier
Joined: 14 Sep 2003 Posts: 22 Location: Montréal QC Canada |
Posted: Wed Oct 08, 2003 12:00 pm Post subject: |
|
|
Alain, Rob, Dan,
I have the last year's worth of messages (approx 13000) in a Eudora mailbox file. I can contribute it if it's of any use.
Regards
JCT |
|
Back to top |
|
|
The Robman Site Owner
Joined: 01 Aug 2003 Posts: 21271 Location: Chicago, IL |
Posted: Wed Oct 08, 2003 12:25 pm Post subject: |
|
|
Thanks JC, but I'm not sure what I could do with it, any ideas? _________________ Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help! |
|
Back to top |
|
|
Mark Pierson Expert
Joined: 03 Aug 2003 Posts: 3017 Location: Connecticut, USA |
Posted: Wed Oct 08, 2003 5:56 pm Post subject: |
|
|
The Robman wrote: | any ideas? |
Not an elegant solution, but if there's no easy way to get the old messages rolled in over here, why not create some sort of text file digest and upload it to the Files area? Another option might be a large HTML placed at hifi-remote. _________________ Mark |
|
Back to top |
|
|
gfb107 Expert
Joined: 03 Aug 2003 Posts: 3411 Location: Cary, NC |
|
Back to top |
|
|
The Robman Site Owner
Joined: 01 Aug 2003 Posts: 21271 Location: Chicago, IL |
Posted: Wed Oct 08, 2003 8:13 pm Post subject: |
|
|
I know that I don't have the time to look into this sort of thing myself, but if someone else were to do all the hard work and create a zip file for me, I'd be more than happy to host it. _________________ Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help! |
|
Back to top |
|
|
Dabbith
Joined: 04 Aug 2003 Posts: 55 Location: Anonia, CT |
Posted: Thu Oct 09, 2003 7:37 am Post subject: |
|
|
Well, I'm making some progress. I was able to use the yahoo2mbox script to download all 29663 messages to an mbox file (66mb). Now I just need to see if there's an easy way to bulk import it into phpbb. I know that there are scripts out there to convert it to a set of HTML files (hypermail), but it would be really nice if it was a searchable part of the board. If I don't find way to get it into phpBB in a couple of days, I'll create the HTML files and send them to Rob.
Thanks for finding the script Alain, it worked great. |
|
Back to top |
|
|
aberguerand Advanced Member
Joined: 11 Aug 2003 Posts: 257 Location: Lausanne, VD, Switzerland |
Posted: Thu Oct 09, 2003 10:33 am Post subject: |
|
|
I also tried to scrap a couple of messages with yahoo2mbox and it worked like a charm.
I surfed through the phpBB forums to look for ways of importing the Yahoo forum into the phpBB one. I did not find a simple way to do it. By simple, i mean simple for someone like me, that has no previous knowledge of php or the administration of phpBB boards. Basicall, the solutions I found either require adapting php scripts (the complexity of which I cannot evaluate) to the data structure we want to import or generating sql scripts that must then be directly imported (risky) in the underlying database. But maybe a php guru might help...
I also analysed the generate MBOX file, and it contains extensive information on the messages, including the "In-Reply-To:" that could be used to regenerate the threads. I also noticed that all the email addresses are in clear, unobfuscated form, so they would need to be somehow altered before the whole archive is put on-line.
Alain |
|
Back to top |
|
|
Dabbith
Joined: 04 Aug 2003 Posts: 55 Location: Anonia, CT |
Posted: Fri Nov 14, 2003 4:28 pm Post subject: |
|
|
Rob,
Sorry it's taken so long, I've been a bit busy and a bit lazy. I've finally compiled Hypermail, and created a set of pages with all of the JP1 Yahoo group messages. Uncompressed they take up 186MB (29,123 files). If that's not going to be too big, I'll compress them and put the compressed file on a web server and give you a link to it.
Dan |
|
Back to top |
|
|
|