Unix local path syntax to limit data traffic?

gnfti · Posted: Wed Apr 02, 2008 11:56 am

Hi all,

I run Electicker, which uses a truckload of FFA scripts to gather data. To limit page load times I cache everything using a Ruby script that runs every few minutes, like so:

Code:

data = open("http://www.electicker2008.com/php/rss2html2.php?XMLFILE=http://www.electicker2008.com/php/filter.php?feed=http://www.electicker2008.com/php/mesh-headlines.php&TEMPLATE=http://www.electicker2008.com/php/news-relative-template.html&NOFUTUREITEMS=1&MAXITEMS=8&ItemDescriptionLength=100").read
File.open(this_directory + "/headlines.html", 'w') do |f|
f.write data
end

You will notice the multiple steps it calls to render the data, i.e. Mesh -> Filter -> RSS2HTML.

Now, because the URLs here are all in "http://" format, it generates a huge amount of data, which counts towards (and well over) my traffic quota. (Electicker runs more than 20 versions of the above code.)

My web host wants me to change the URLs to local paths, i.e.

Code:

/srv/customers/[$MYCUSTNO]/electicker2008.com/www/php/[$FILE]

How do I format this in the Ruby script so it works? I'm getting 500 errors right now. Thanks in advance for any help.

MacSupport · Joined: 24 Feb 2005 Posts: 2805

gnfti wrote:

Hi all,

I run Electicker, which uses a truckload of FFA scripts to gather data. To limit page load times I cache everything using a Ruby script that runs every few minutes, like so:

Code:

data = open("http://www.electicker2008.com/php/rss2html2.php?XMLFILE=http://www.electicker2008.com/php/filter.php?feed=http://www.electicker2008.com/php/mesh-headlines.php&TEMPLATE=http://www.electicker2008.com/php/news-relative-template.html&NOFUTUREITEMS=1&MAXITEMS=8&ItemDescriptionLength=100").read
File.open(this_directory + "/headlines.html", 'w') do |f|
f.write data
end

You will notice the multiple steps it calls to render the data, i.e. Mesh -> Filter -> RSS2HTML.

Now, because the URLs here are all in "http://" format, it generates a huge amount of data, which counts towards (and well over) my traffic quota. (Electicker runs more than 20 versions of the above code.)

My web host wants me to change the URLs to local paths, i.e.

Code:

/srv/customers/[$MYCUSTNO]/electicker2008.com/www/php/[$FILE]

How do I format this in the Ruby script so it works? I'm getting 500 errors right now. Thanks in advance for any help.

It looks like:

http://www.electicker2008.com/php/rss2html2.php

Would become:

/srv/customers/[$MYCUSTNO]/electicker2008.com/www/php/rss2html2.php

But that is just what I think.

Jim

gnfti · Posted: Sun Apr 20, 2008 4:31 pm

Thanks for the reply. As I mentioned in my original post I had already tried that: turns out, the problem was the multiple iterations (i.e. mesh->filter->rss2html). You could run the RSS2HTML locally, but it still expects "http://..." for the filter and mesh scripts, and refuses to run otherwise.

I was unable to solve this with the given scripts. As a workaround, I moved all the PHP action to my home server and made it run and upload (FTP) via pycron and ruby.

It would be great if all the FFA scripts could be adapted to accept local Unix paths.

For cross-reference, a discussion of my situation on Ask Metafilter.

MacSupport · Joined: 24 Feb 2005 Posts: 2805

gnfti wrote:

Thanks for the reply. As I mentioned in my original post I had already tried that: turns out, the problem was the multiple iterations (i.e. mesh->filter->rss2html). You could run the RSS2HTML locally, but it still expects "http://..." for the filter and mesh scripts, and refuses to run otherwise.

I was unable to solve this with the given scripts. As a workaround, I moved all the PHP action to my home server and made it run and upload (FTP) via pycron and ruby.

It would be great if all the FFA scripts could be adapted to accept local Unix paths.

For cross-reference, a discussion of my situation on Ask Metafilter.

Maybe this will help. It can run just fine from shell scripts. Below is an extract from my testing script:

Code:

testnum=1
testAttempt=1
testPassed=0
testdescr="TEST #"$testnum": Simple UTF-8"
#
cp ../../rss2html.php ../../FeedForAll_XMLParser.inc.php .
cp ../template1.html sample-template.html
cp ../feed1.xml .
php -r '$XMLFILE="feed1.xml"; include("rss2html.php");' > actual$testnum.txt
echo -n $testdescr
diff -u ../expect$testnum.txt actual$testnum.txt > /dev/null
if [ $? -eq 0 ]; then
testPassed=$(($testPassed+1))
echo " Passed"
rm -rf *.rsp FeedForAllCacheFiles rssFilterCacheFiles rssMeshCacheFiles sql2rs
sCacheFiles tmp$testnum.msg expect$testnum.txt actual$testnum.txt *.php *.xml *.
html
else
echo " FAILED"
fi

Jim