FeedForAll Forum Index  
 Home  •  FAQ  •   Search   •  Register   •  Profile   •  Log in
 Unix local path syntax to limit data traffic? View next topic
View previous topic
This forum is locked: you cannot post, reply to, or edit topics.This topic is locked: you cannot edit posts or make replies.
Author Message
gnfti



Joined: 15 Jan 2008
Posts: 15
Location: The Hague, Netherlands

PostPosted: Wed Apr 02, 2008 11:56 am Reply with quoteBack to top

Hi all,

I run Electicker, which uses a truckload of FFA scripts to gather data. To limit page load times I cache everything using a Ruby script that runs every few minutes, like so:

Code:
data = open("http://www.electicker2008.com/php/rss2html2.php?XMLFILE=http://www.electicker2008.com/php/filter.php?feed=http://www.electicker2008.com/php/mesh-headlines.php&TEMPLATE=http://www.electicker2008.com/php/news-relative-template.html&NOFUTUREITEMS=1&MAXITEMS=8&ItemDescriptionLength=100").read
       File.open(this_directory + "/headlines.html", 'w') do |f|
               f.write data
       end


You will notice the multiple steps it calls to render the data, i.e. Mesh -> Filter -> RSS2HTML.

Now, because the URLs here are all in "http://" format, it generates a huge amount of data, which counts towards (and well over) my traffic quota. (Electicker runs more than 20 versions of the above code.)

My web host wants me to change the URLs to local paths, i.e.

Code:
/srv/customers/[$MYCUSTNO]/electicker2008.com/www/php/[$FILE]


How do I format this in the Ruby script so it works? I'm getting 500 errors right now. Thanks in advance for any help.
View user's profileVisit poster's website
MacSupport



Joined: 24 Feb 2005
Posts: 2805

PostPosted: Sun Apr 20, 2008 8:27 am Reply with quoteBack to top

gnfti wrote:
Hi all,

I run Electicker, which uses a truckload of FFA scripts to gather data. To limit page load times I cache everything using a Ruby script that runs every few minutes, like so:

Code:
data = open("http://www.electicker2008.com/php/rss2html2.php?XMLFILE=http://www.electicker2008.com/php/filter.php?feed=http://www.electicker2008.com/php/mesh-headlines.php&TEMPLATE=http://www.electicker2008.com/php/news-relative-template.html&NOFUTUREITEMS=1&MAXITEMS=8&ItemDescriptionLength=100").read
       File.open(this_directory + "/headlines.html", 'w') do |f|
               f.write data
       end


You will notice the multiple steps it calls to render the data, i.e. Mesh -> Filter -> RSS2HTML.

Now, because the URLs here are all in "http://" format, it generates a huge amount of data, which counts towards (and well over) my traffic quota. (Electicker runs more than 20 versions of the above code.)

My web host wants me to change the URLs to local paths, i.e.

Code:
/srv/customers/[$MYCUSTNO]/electicker2008.com/www/php/[$FILE]


How do I format this in the Ruby script so it works? I'm getting 500 errors right now. Thanks in advance for any help.


It looks like:

http://www.electicker2008.com/php/rss2html2.php

Would become:

/srv/customers/[$MYCUSTNO]/electicker2008.com/www/php/rss2html2.php

But that is just what I think.

Jim

_________________
Be sure to check out the FeedForAll Mac tutorials at: http://www.FeedForAll.com/tutorials/

FeedForAll Mac and rss2html.php
support@FeedForAll.com
View user's profileVisit poster's website
gnfti



Joined: 15 Jan 2008
Posts: 15
Location: The Hague, Netherlands

PostPosted: Sun Apr 20, 2008 4:31 pm Reply with quoteBack to top

Thanks for the reply. As I mentioned in my original post I had already tried that: turns out, the problem was the multiple iterations (i.e. mesh->filter->rss2html). You could run the RSS2HTML locally, but it still expects "http://..." for the filter and mesh scripts, and refuses to run otherwise.

I was unable to solve this with the given scripts. As a workaround, I moved all the PHP action to my home server and made it run and upload (FTP) via pycron and ruby.

It would be great if all the FFA scripts could be adapted to accept local Unix paths.

For cross-reference, a discussion of my situation on Ask Metafilter.
View user's profileVisit poster's website
MacSupport



Joined: 24 Feb 2005
Posts: 2805

PostPosted: Wed Apr 23, 2008 7:58 pm Reply with quoteBack to top

gnfti wrote:
Thanks for the reply. As I mentioned in my original post I had already tried that: turns out, the problem was the multiple iterations (i.e. mesh->filter->rss2html). You could run the RSS2HTML locally, but it still expects "http://..." for the filter and mesh scripts, and refuses to run otherwise.

I was unable to solve this with the given scripts. As a workaround, I moved all the PHP action to my home server and made it run and upload (FTP) via pycron and ruby.

It would be great if all the FFA scripts could be adapted to accept local Unix paths.

For cross-reference, a discussion of my situation on Ask Metafilter.


Maybe this will help. It can run just fine from shell scripts. Below is an extract from my testing script:

Code:

testnum=1
testAttempt=1
testPassed=0
testdescr="TEST #"$testnum": Simple UTF-8"
#
cp ../../rss2html.php ../../FeedForAll_XMLParser.inc.php .
cp ../template1.html sample-template.html
cp ../feed1.xml .
php -r '$XMLFILE="feed1.xml"; include("rss2html.php");' > actual$testnum.txt
echo -n $testdescr
diff -u ../expect$testnum.txt actual$testnum.txt > /dev/null
if [ $? -eq 0 ]; then
  testPassed=$(($testPassed+1))
  echo " Passed"
  rm -rf *.rsp FeedForAllCacheFiles rssFilterCacheFiles rssMeshCacheFiles sql2rs
sCacheFiles tmp$testnum.msg expect$testnum.txt actual$testnum.txt *.php *.xml *.
html
else
  echo " FAILED"
fi


Jim

_________________
Be sure to check out the FeedForAll Mac tutorials at: http://www.FeedForAll.com/tutorials/

FeedForAll Mac and rss2html.php
support@FeedForAll.com
View user's profileVisit poster's website
Display posts from previous:      
This forum is locked: you cannot post, reply to, or edit topics.This topic is locked: you cannot edit posts or make replies.


 Jump to:   



View next topic
View previous topic


Powered by phpBB © 2001, 2002 phpBB Group :: FI Theme