1. Have you installed the new JSF Mobile app? Check out all the details here.
    Dismiss Notice
  2. One account & one avatar for all of JSF. Unified login and profile. Forum alerts on the main site, and more. Check out the details here: Forum & main site unified account feature is live!
    Dismiss Notice

JSF Main site redesign and other new features

Discussion in 'Technical' started by John Stone, Aug 15, 2009.

  1. Pawn

    Pawn Active Member

    Joined:
    Dec 31, 2008
    Messages:
    21
    Likes Received:
    0
    John,

    You mentioned in your daily update today that the task of porting over all of your previous news updates is going to be a monumental task.

    I don't know how you have things stored on the backend, but there might be an easier way to do it rather than by hand. If all of your daily updates are in a database, it would be much easier to write some scripts to select, transform, and reinsert the data into the WordPress database than to do it all by hand.

    If they are just pasted statically onto php pages :-)eek:) it could be possible to write a web scraping script to at least do some consolidating and organizing of the data before trying to fit it into WordPress. http://www.google.com/search?q=free+web+scraper+program
     
  2. Reno_1ted

    Reno_1ted Well-Known Member

    Joined:
    Mar 15, 2004
    Messages:
    1,956
    Likes Received:
    1
    John,

    Any reason you cannot just link to the old updates in their current format, until you port them all across, that would buy you more time? :)
     
  3. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    :nod: :o

    Unfortunately each entry is going to have to be "massaged" by a human: every link and every image will be broken and will need to be fixed by hand, and I'll also need to add tags, an SEO-friendly title and manually enter the date of the post. The actual copy and paste of each entry will only take a moment, it's the rest of the stuff that will take forever. I don't know of any way to automate these tasks, unfortunately. :(
     
  4. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    That's definitely something I'm considering. My concern is if I do that I will procrastinate. :o
     
  5. gazareth

    gazareth Senior Member

    Joined:
    Aug 1, 2006
    Messages:
    6,162
    Media:
    2
    Albums:
    1
    Likes Received:
    52
    It might take some serious PHP-ninjaing, but I think it might be possible.

    You have all the news updates on a PHP file for each month. You could setup a PHP script to scrape one of these files at a time, doing the following:
    *Scraping the content of the post and dumping it into the appropriate word press DB field (easy enough if you have been consistent with formatting)
    *Scraping the date which heads the post (again, if formatting is consistent this should be possible) and then using PHP/MySQL to convert it into the appropriate date/time stamp format for the WP database. Scrape all the posts into an array... reverse the array so that the key # is then the same as the post's date... you will already know the month and year from the PHP file and can pre-input it.
    *For images, could you just run a simple find/replace script? e.g. dump all the old images into /oldimages/ and then find/replace the URLs to point at that folder?
    *Links should be do-able too, find/replace. The only links that I can think of that would be really awkard, if not impossible, would be links to other news updates.

    :confused:
     
  6. Pawn

    Pawn Active Member

    Joined:
    Dec 31, 2008
    Messages:
    21
    Likes Received:
    0
    I think a lot of this can be automated to a degree. Pulling out the post, the date, fixing links.. those should all be fixable with a well written script. Going back through and tagging, SEO-maximizing, etc would be a heck of a chore though. You could maybe set up a system where if a post contains a keyword, it adds that as a tag.

    Of course you could also spend all day designing a sweet system to go through and pluck all of your old posts that would take as long to implement as it would to do it by hand.. :confused:

    Regarding the sheer volume of posts, don't forget that you have a large community behind you that could help with some of this... think Folding@Home. :cool:
     
  7. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    Gaz, Pawn... you guys make some really good points. Perhaps I'm being too hasty. Even if it's not perfect, a script could save me a lot of time. I will put some time into researching the possibilities over the weekend.

    Thanks a lot! :)
     
  8. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    :bang:

    I've been working all morning on scraping the daily news updates. I've had some measure of success, mainly with the dates and titles, but the actual updates are tough to key in on reliably.

    I also looked at the WP database and I have some concerns about manually inserting posts: specifically the unique guid and the post_name field. Has anyone had any luck directly inserting a large number WP posts into the database?

    I'm outside of my comfort zone here, and don't want to corrupt the database.

    Anyway, I don't think I've got the skills to pull this one off.
     
  9. gazareth

    gazareth Senior Member

    Joined:
    Aug 1, 2006
    Messages:
    6,162
    Media:
    2
    Albums:
    1
    Likes Received:
    52
    Does the guid not automatically increment itself when you put in a new post?

    Backup the database, then try inserting something without putting anything as the guid.

    For the post name, could you use the WP web admin panel to insert a post and see how it handles that field by looking at the raw DB data afterwards?
     
  10. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    Nope, this is generated by WP when you add a post through "normal" channels. The only auto-increment is the "id" field, which is not the same.


    I can tell that anyway by looking at the DB. It's null. It would have to be added just like the guid. My concern is WP would not be aware of the manual additions, and would dup these values, causing life as we know it to end ( I suppose I could dig into the code and find out where this counter is kept in the DB).
     
  11. gazareth

    gazareth Senior Member

    Joined:
    Aug 1, 2006
    Messages:
    6,162
    Media:
    2
    Albums:
    1
    Likes Received:
    52
    If you could find the SQL query in the WP PHP files then that would probably hold the answer.
     
  12. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    Yeah, I can grep the code for that. My bigger concern is I'm overlooking something small that will cascade after I've done all the work and ruin everything. That's why I am hoping to hear from someone who's done something like this before.

    :scared:

    Of course isolating the daily updates for a good scrape seems all but impossible anyway.
     
  13. gravityhomer

    gravityhomer Elite Member
    Lifetime Platinum Member

    Joined:
    Jan 23, 2004
    Messages:
    3,609
    Likes Received:
    3
    I have no web design expertise but have done a lot of coding for data manipulation. If you are having trouble getting one database software to talk to another database software is it possible you could use a real simple intermediary?

    When I am having trouble like this, I always try to make something a simple .txt file. Anyway you can export a giant txt file with every news update?

    Then read that giant txt file into the new database?

    I am completely aware this advice may be of no hope.
    :gl:
     
  14. boomboom

    boomboom Well-Known Member

    Joined:
    Feb 6, 2005
    Messages:
    83
    Likes Received:
    0
    John - contact me. I'll help you with the automation. :)
     
  15. jk0

    jk0 Elite Member
    Lifetime Platinum Member

    Joined:
    Feb 13, 2004
    Messages:
    7,509
    Likes Received:
    4
    When you scrape all of the data just turn it into an XML file. WP lets you import different kinds of feeds from within the admin panel.
     
  16. Pawn

    Pawn Active Member

    Joined:
    Dec 31, 2008
    Messages:
    21
    Likes Received:
    0
    Haven't heard any news on this lately in the daily news updates. BodyShop is looking nice but how is progress coming with the site-wide redesign, particularly your archiving of the old news items?
     
  17. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,867
    Likes Received:
    75
    Slowly but surely. My job has me working a lot of hours right now, and lately I've been spending most of my off time out biking.

    The daily news archives are still a major dilemma.
     

Share This Page