JP1 Remotes Forum Index JP1 Remotes


FAQFAQ SearchSearch 7 days of topics7 Days MemberlistMemberlist UsergroupsUsergroups RegisterRegister
ProfileProfile Log in to check your private messagesLog in to check your private messages Log inLog in

Historical data
Goto page 1, 2  Next
 
Post new topic   Reply to topic    JP1 Remotes Forum Index -> Web Site Issues
View previous topic :: View next topic  
Author Message
Haggis



Joined: 14 Sep 2003
Posts: 27
Location: Brisbane

                    
PostPosted: Wed Oct 01, 2003 5:25 pm    Post subject: Historical data Reply with quote

There is a lot of interesting and useful information that is still at the old Yahoo JP1 group. Is there an easy way to migrate useful posts to this forum?

Haggis
Question
Back to top
View user's profile Send private message
Mark Pierson
Expert


Joined: 03 Aug 2003
Posts: 3017
Location: Connecticut, USA

                    
PostPosted: Wed Oct 01, 2003 6:42 pm    Post subject: Reply with quote

Yahoo doesn't make it easy to do anything. Mad

There's no way that I've found (at the admin level) to get any message content out of the Yahoo group so that it could be rolled in over here. That's why Rob has said the old group will remain intact as long as necessary to serve as an archive. While obviously not the best solution, it beats losing all that history.

Of course, if you're feeling ambitious, you could open EVERY message over there and copy & paste it into a new one over here! Shocked
_________________
Mark
Back to top
View user's profile Send private message Send e-mail Visit poster's website
The Robman
Site Owner


Joined: 01 Aug 2003
Posts: 21211
Location: Chicago, IL

                    
PostPosted: Wed Oct 01, 2003 7:35 pm    Post subject: Reply with quote

Hey there Haggis, I thank you for stepping forward and offering your services. Please feel free to go through the Yahoo archives and identify all the posts that you think are useful and list their MSG# here and someone will copy them.

I am joking of course, unless of course you're just crazy enough to try actually doing that. In which case, please report back towards the end of next year and let us know your progress! Smile
_________________
Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help!
Back to top
View user's profile Send private message Visit poster's website
Dabbith



Joined: 04 Aug 2003
Posts: 55
Location: Anonia, CT

                    
PostPosted: Mon Oct 06, 2003 9:45 am    Post subject: Reply with quote

Rob,
Thinking about the historical data...

I've actually got the messages from 6/6/2002 on archived into an outlook data file. I'm sure there are others that have similar archives. I don't think it would be too much trouble to spend a bit of time writing a macro to try and post these messages, but you'd need to set up a special section for them. I'd also need to know if anyone had the earlier posts.

Another option that might be worth investigating would be for me to write a macro that attempted to harvest the data from Yahoo and then add it to this site, that way we could have all of the posts here.

One of the problems would be that all of the posts would appear to come from me. I would at least include the original poster and date posted in the body.

Is there any way to add posts in a bulk format?

If you're interested in this idea, I'll see if I can come up with a way to scrape the data from Yahoo.

Dan
Back to top
View user's profile Send private message
The Robman
Site Owner


Joined: 01 Aug 2003
Posts: 21211
Location: Chicago, IL

                    
PostPosted: Mon Oct 06, 2003 10:12 am    Post subject: Reply with quote

Hey Dan,
Insteresting idea. Would your macros be able to join posts together is thread format?

As for what's possible with these phpbb forums, I'm not entirely sure. If you want to do some research, go to the http://www.phpbb.com web site and take a look.

What might be easier would be to have your macros create regular HTML files with the correct user ids, etc, then I could just load these up as regular web site pages.
_________________
Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help!
Back to top
View user's profile Send private message Visit poster's website
Dabbith



Joined: 04 Aug 2003
Posts: 55
Location: Anonia, CT

                    
PostPosted: Wed Oct 08, 2003 9:16 am    Post subject: Reply with quote

I started to look into it a bit. Apperently, there are converters from several other forums to phpBB. Unfortunatly, Yahoo is not one of them. Also unfortunate is that all of them are written in PHP which I don't really have time to learn right now. There was some talk of trying to create a converter for Yahoo in March, but it seems to have died off. I think I'm going to try and scrape the data from yahoo into a database. If I manage that, I'll re-visit getting the data to this forum.

Is there anyone around who knows php?

If we could find someone, they might be able to create/modify a converter to import the database data.
Back to top
View user's profile Send private message
aberguerand
Advanced Member


Joined: 11 Aug 2003
Posts: 257
Location: Lausanne, VD, Switzerland

                    
PostPosted: Wed Oct 08, 2003 10:34 am    Post subject: Reply with quote

There is a tool (Perl script) that can scrap the contents of a Yahoo Group and save it as an Unix email archive :

http://www.lpthe.jussieu.fr/~zeitlin/yahoo2mbox.html

There seems however that Yahoo places a limit on the quantity of data one user can download for a given period of time, so the whole archive might need to be get in several passes, and/or in a coordinated effort between several users.
Back to top
View user's profile Send private message
JCTerrier



Joined: 14 Sep 2003
Posts: 22
Location: Montréal QC Canada

                    
PostPosted: Wed Oct 08, 2003 12:00 pm    Post subject: Reply with quote

Alain, Rob, Dan,

I have the last year's worth of messages (approx 13000) in a Eudora mailbox file. I can contribute it if it's of any use.

Regards
JCT
Back to top
View user's profile Send private message
The Robman
Site Owner


Joined: 01 Aug 2003
Posts: 21211
Location: Chicago, IL

                    
PostPosted: Wed Oct 08, 2003 12:25 pm    Post subject: Reply with quote

Thanks JC, but I'm not sure what I could do with it, any ideas?
_________________
Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help!
Back to top
View user's profile Send private message Visit poster's website
Mark Pierson
Expert


Joined: 03 Aug 2003
Posts: 3017
Location: Connecticut, USA

                    
PostPosted: Wed Oct 08, 2003 5:56 pm    Post subject: Reply with quote

The Robman wrote:
any ideas?

Not an elegant solution, but if there's no easy way to get the old messages rolled in over here, why not create some sort of text file digest and upload it to the Files area? Another option might be a large HTML placed at hifi-remote.
_________________
Mark
Back to top
View user's profile Send private message Send e-mail Visit poster's website
gfb107
Expert


Joined: 03 Aug 2003
Posts: 3411
Location: Cary, NC

                    
PostPosted: Wed Oct 08, 2003 6:43 pm    Post subject: Reply with quote

Failing the ability to actually put it all into these forums, I would suggest converting it to a set of HTML files that are formatted as much as possible to look like this forum. Then maybe Rob could host it as http://www.hifi-remote.com/archive/* or something.

I'm pretty good with HTML and scripting, so I could take a stab at converting the email log, but it would take time away from RM development. Confused
_________________
-- Greg
Original RemoteMaster developer
JP1 How-To's and Software Tools
The #1 Code Search FAQ and it's answer (PLEASE READ FIRST)
Back to top
View user's profile Send private message Visit poster's website
The Robman
Site Owner


Joined: 01 Aug 2003
Posts: 21211
Location: Chicago, IL

                    
PostPosted: Wed Oct 08, 2003 8:13 pm    Post subject: Reply with quote

I know that I don't have the time to look into this sort of thing myself, but if someone else were to do all the hard work and create a zip file for me, I'd be more than happy to host it.
_________________
Rob
www.hifi-remote.com
Please don't PM me with remote questions, post them in the forums so all the experts can help!
Back to top
View user's profile Send private message Visit poster's website
Dabbith



Joined: 04 Aug 2003
Posts: 55
Location: Anonia, CT

                    
PostPosted: Thu Oct 09, 2003 7:37 am    Post subject: Reply with quote

Well, I'm making some progress. I was able to use the yahoo2mbox script to download all 29663 messages to an mbox file (66mb). Now I just need to see if there's an easy way to bulk import it into phpbb. I know that there are scripts out there to convert it to a set of HTML files (hypermail), but it would be really nice if it was a searchable part of the board. If I don't find way to get it into phpBB in a couple of days, I'll create the HTML files and send them to Rob.

Thanks for finding the script Alain, it worked great.
Back to top
View user's profile Send private message
aberguerand
Advanced Member


Joined: 11 Aug 2003
Posts: 257
Location: Lausanne, VD, Switzerland

                    
PostPosted: Thu Oct 09, 2003 10:33 am    Post subject: Reply with quote

I also tried to scrap a couple of messages with yahoo2mbox and it worked like a charm.

I surfed through the phpBB forums to look for ways of importing the Yahoo forum into the phpBB one. I did not find a simple way to do it. By simple, i mean simple for someone like me, that has no previous knowledge of php or the administration of phpBB boards. Basicall, the solutions I found either require adapting php scripts (the complexity of which I cannot evaluate) to the data structure we want to import or generating sql scripts that must then be directly imported (risky) in the underlying database. But maybe a php guru might help...

I also analysed the generate MBOX file, and it contains extensive information on the messages, including the "In-Reply-To:" that could be used to regenerate the threads. I also noticed that all the email addresses are in clear, unobfuscated form, so they would need to be somehow altered before the whole archive is put on-line.

Alain
Back to top
View user's profile Send private message
Dabbith



Joined: 04 Aug 2003
Posts: 55
Location: Anonia, CT

                    
PostPosted: Fri Nov 14, 2003 4:28 pm    Post subject: Reply with quote

Rob,

Sorry it's taken so long, I've been a bit busy and a bit lazy. I've finally compiled Hypermail, and created a set of pages with all of the JP1 Yahoo group messages. Uncompressed they take up 186MB (29,123 files). If that's not going to be too big, I'll compress them and put the compressed file on a web server and give you a link to it.

Dan
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic       JP1 Remotes Forum Index -> Web Site Issues All times are GMT - 5 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

Powered by phpBB © 2001, 2005 phpBB Group
Top 7 Advantages of Playing Online Slots The Evolution of Remote Control