Sunday, March 20, 2011

Bandwidth Optimization with FreeBSD 7 + Squid + Apache + videocache


Bismillahirrahmanirrahim

Web technology that is increasingly growing not balanced with bandwidth prices down very slowly. It certainly will make us as administrators dizzy 7 circumference on the other side must conserve bandwidth expenditures, on the other hand very often users access the streaming video which is very wasteful of bandwidth.The most difficult of access with limited bandwidth there is video streaming, an increasingly common around us with flash technology
with Flv file format. Besides youtube many other websites, which provides streaming video of them google video, meta cafe.
There's an interesting idea from the people who joined in cachevideos.com to mencache video video of frequently accessed. they called the application with the name videocache.


What is Videocache?
videochache Squid is an application dicoding URL rewriter plugin using Python to optimize bandwidth when accessing web site (portal) video sharing like Youtube, Metacafe etc. This application will help you save bandwidth when access of video from the same network to some videos which are often in request more than once.

Is not enough video caching with squid?
Squid can not the video video caches dynamically. Videocache adjust into the squid to assist caches with a good video. Video is in the cache are stored into the local disk of your server and you can access the video like you have downloaded. You also can copy the video video to another server with ease.

How does this videocache?
when a video at the request of a website or portal, videocache who requested the video store into a proxy server hard drive. And when there are new requests the same video that has saved the accessor to access the video directly from the proxy server (as though from youtube. Note: if access youtube). This will really help save bandwidth and speed up loading of the video.

Where we can get this application?
Videocahce is the Free Application for various platforms and operating systems, and can be downloaded from his site that is http://cachevideos.com/download

What is needed to make this Videocache?

Software required:
1. FreeBSD 7.0-RELEASE
2. Squid> = 2.6
3. Python> = 2.4
4. Python-urlgrabber
5. Python-iniparse
6. Apache
7. Videocache

How where his menginstallasi way?

The way the installation is relatively easy because "there is technology to make human life easier not even become more difficult" so do not live

This easy you make it difficult (Learning that will make human life easier, never stop to learn if it did not want to suffer).
Okay let's start this installation stage:

1. You can install FreeBSD 7 With good and true (for those who can not install FreeBSD it is better you learn it first before moving to make this videocahe)

2. Install Squid
login to your server with root privileges
indofreebsd# cd /usr/ports/www/squid
indofreebsd# make install clean
Squid Configuration
indofreebsd# nano /usr/local/etc/squid/squid.conf
----isi squid.conf--------
http_port 3128 transparent
cache_effective_user squid
cache_effective_group squid
acl all src 0.0.0.0/0.0.0.0
icp_query_timeout 1000
high_memory_warning 500 MB
visible_hostname zorro.indofreebsd.or.id
httpd_suppress_version_string on
cache_mem 64 MB
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_swap_low 90
cache_swap_high 95
maximum_object_size 131072 KB
maximum_object_size_in_memory 64 KB
tcp_recv_bufsize 65535 bytes
ipcache_size 8192
fqdncache_size 8192
acl msnmess url_regex http://207.46.111.55/gateway/gateway.dll?
deny_info TCP_RESET msnmess
http_access deny msnmess
forwarded_for off
high_page_fault_warning 10
high_response_time_warning 2000
client_persistent_connections off
server_persistent_connections on
half_closed_clients off
cache_dir aufs /cache 10000 10 256
log_icp_queries off
cache_access_log /usr/local/squid/logs/access.log
emulate_httpd_log on
ftp_user team@indofreebsd.or.id This e-mail address is being protected from spambots. You need JavaScript enabled to view it
cache_mgr team@indofreebsd.or.id This e-mail address is being protected from spambots. You need JavaScript enabled to view it
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
quick_abort_min 0 KB
quick_abort_max 0 KB
negative_dns_ttl 2 minutes
acl mynetwork src 192.168.1.0/26
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl SSL_ports port 443 563 2082 2083 2086 2087 2093 2095 2096
acl Safe_ports port 80 21 443 563 70 210 8000 11999 2082 2083 2086 2087 2095 2096 8082 8090
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow mynetwork
http_access allow localhost
deny_info TCP_RESET all
http_access deny all
snmp_port 3001
acl queryme snmp_community SquidSnmpRahasia
acl adminpc src 192.168.1.2
snmp_access allow queryme localhost
snmp_access allow queryme adminpc
snmp_access deny all
icp_access allow mynetwork
icp_access deny all
miss_access allow all
ie_refresh on

----squid.conf-------
save with the name squid.conf

Next create a swap directory

indofreebsd # squid-z
05/10/2009 16:38:37 | Creating Swap Directories

let the road every time restart the input parameter to the / etc / rc.conf
squid_enable = "YES"
3. Install Python


indofreebsd # cd / usr / ports / lang / python
indofreebsd # make install clean


4. Install Python-urlgrabber
first download its source in http://linux.duke.edu/projects/urlgrabber/download/urlgrabber-3.1.0.tar.gz or http://www.indofreebsd.or.id/download/urlgrabber-3.1.0 . tar.gz




indofreebsd # fetch http://linux.duke.edu/projects/urlgrabber/download/urlgrabber-3.1.0.tar.gz
indofreebsd # tar-zxvf urlgrabber-3.1.0.tar.gz
indofreebsd-3.1.0 # cd urlgrabber
indofreebsd # python setup.py install


5. Install Python-iniparse
first download its source in http://iniparse.googlecode.com/files/iniparse-0.3.1.tar.gz or in http://www.indofreebsd.or.id/download/iniparse-0.3.1.tar. gz

indofreebsd # fetch http://iniparse.googlecode.com/files/iniparse-0.3.1.tar.gz
indofreebsd # tar-zxvf iniparse-0.3.1.tar.gz
indofreebsd-0.3.1 # cd iniparse
indofreebsd # python setup.py install


6. Install Apache

indofreebsd # cd / usr/ports/www/apache22
indofreebsd # make install clean
let the road every time restart the input parameter to the / etc / rc.conf
apache22_enable = "YES"


7. Install Videocache
first download its source in http://www.indofreebsd.or.id/download/videocache-1.9.1.tar.gz


indofreebsd # fetch http://www.indofreebsd.or.id/download/videocache-1.9.1.tar.gz
indofreebsd # tar-zxvf videocache-1.9.1.tar.gz
indofreebsd# cd videocache-1.9.1
indofreebsd# python setup.py install
indofreebsd# cp videocache-httpd.conf /usr/local/etc/apache22/videocache.conf
indofreebsd# cp videocache-sysconfig.conf /etc/videocache.conf
indofreebsd# vi /etc/videocache.conf
Video Cache Configuration
#----- Content videocache.conf -----
enable_video_cache = 1
cache_host = 192.168.0.1
proxy = http://192.168.0.1:3128/
hit_threshold = 1
base_dir = / usr/local/www/apache22/data/videocache /
disk_avail_threshold = 100
temp_dir = tmp
max_parallel_downloads = 30
enable_videocache_cleaner = 1
video_lifetime = 60
logdir = / usr / local / squid / logs /
max_logfile_size = 10
max_logfile_backups = 10
rpc_host = 127.0.0.1
rpc_port = 9100
enable_youtube_cache = 1
youtube_cache_dir = youtube
max_youtube_video_size = 0
min_youtube_video_size = 0

# Metacafe.com Options
enable_metacafe_cache = 1
metacafe_cache_dir = Metacafe
max_metacafe_video_size = 0
min_metacafe_video_size = 0

# Dailymotion.com Options
enable_dailymotion_cache = 1
dailymotion_cache_dir = Dailymotion
max_dailymotion_video_size = 0
min_dailymotion_video_size = 0

# Google.com Options
enable_google_cache = 1
google_cache_dir = google
max_google_video_size = 0
min_google_video_size = 0

# Redute.com Options
enable_redtube_cache = 1
redtube_cache_dir = redtube
max_redtube_video_size = 0
min_redtube_video_size = 0

# Xtube.com Options
enable_xtube_cache = 1
xtube_cache_dir = xtube
max_xtube_video_size = 0
min_xtube_video_size = 0

# Vimeo.com Options
enable_vimeo_cache = 1
vimeo_cache_dir = vimeo
max_vimeo_video_size = 0
min_vimeo_video_size = 0

# Wrzuta.pl Options
enable_wrzuta_cache = 1
wrzuta_cache_dir = wrzuta
max_wrzuta_video_size = 0
min_wrzuta_video_size = 0

# Youporn.com Options
enable_youporn_cache = 1
youporn_cache_dir = YouPorn
max_youporn_video_size = 0
min_youporn_video_size = 0

# Soapbox.msn.com Options
enable_soapbox_cache = 1
soapbox_cache_dir = Soapbox
max_soapbox_video_size = 0
min_soapbox_video_size = 0

# Tube8.com Options
enable_tube8_cache = 1
tube8_cache_dir = tube8
max_tube8_video_size = 0
min_tube8_video_size = 0


# Tvuol.uol.com.br Options
enable_tvuol_cache = 1
tvuol_cache_dir = tvuol
max_tvuol_video_size = 0
min_tvuol_video_size = 0

# Blip.tv Options
enable_bliptv_cache = 1
bliptv_cache_dir = bliptv
max_bliptv_video_size = 0
min_bliptv_video_size = 0

# Break.com Options
enable_break_cache = 1
break_cache_dir = break
max_break_video_size = 0
min_break_video_size = 0
#------ Content videocache.conf arrived here ----#

8. configuring squid.conf squid helper at the store at the bottom of the squid configuration
indofreebsd # vi / usr / local / etc / squid / squid.conf
#----------- Content of squid helper videocache -----#
url_rewrite_program / usr / bin / python / usr / share / videocache / videocache.py
url_rewrite_children 7
url_regex videocache_allow_url acl-i \. youtube \. com \ / get_video \?
url_regex videocache_allow_url acl-i \. googlevideo \. com \ / videoplayback \. googlevideo \. com \ / videoplay \. googlevideo \. com \ / get_video \?
url_regex videocache_allow_url acl-i \. google \. com \ / videoplayback \. google \. com \ / videoplay \. google \. com \ / get_video \?
url_regex videocache_allow_url acl-i \. google \. [az] [az] \ / videoplayback \. google \. [az] [az] \ / videoplay \. google \. [az] [az] \ / get_video \?
acl-i videocache_allow_url url_regex

(25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0 - 4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?

) \ / Videoplayback \?
acl-i videocache_allow_url url_regex

(25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0 - 4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?

) \ / Videoplay \?
acl-i videocache_allow_url url_regex

(25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0 - 4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?) \. (25 [0-5] | 2 [0-4] [0-9] | [01]? [0-9] [0-9]?

) \ / Get_video \?
url_regex videocache_allow_url acl proxy-i [a-Z0-9 \ -] [a-Z0-9] [a-Z0-9] [a-Z0-9]? \. DailyMotion \. com \ /
acl videocache_allow_url url_regex vid-i \. MWA \. DailyMotion \. com \ /
url_regex videocache_allow_url acl-i [a-Z0-9] [0-9a-z] [0-9a-z] [0-9a-z] [0-9a-z]? \. xtube \. com \ flv /(.*)
acl videocache_allow_url url_regex bitcast-i \. vimeo \. com \ / vimeo \ / videos \ /
url_regex videocache_allow_url acl-i va \. wrzuta \. pl \ / wa [0-9] [0-9] [0-9] [0-9]?
url_regex videocache_allow_url acl-i \. files \. YouPorn \. com \ /(.*) \ / flv \ /
url_regex videocache_allow_url acl-i \. msn \. com \. edgesuite \. net \ /(.*) \. flv
acl videocache_allow_url url_regex media-i [a-Z0-9] [a-Z0-9] [a-Z0-9]? \. tube8 \. com \ / mobile [a-Z0-9] [a- Z0-9] [a-Z0-9]? \. tube8 \. com \ /
url_regex videocache_allow_url acl-i \. Mais \. OPM \. com \. br \ /(.*) \. flv
url_regex videocache_allow_url acl-i \. videos [a-Z0-9] [a-Z0-9]? \. blips \. tv \ /(.*) \. (wmv | avi | mov | mp3 | m4v | mp4 | wmv | rm | ram)
acl videocache_allow_url url_regex video-i \. break \. com \ /(.*) \. (flv | mp4)
acl videocache_allow_dom dstdomain. mccont.com dl.redtube.com. cdn.dailymotion.com
url_regex videocache_deny_url acl-i http: \ / \ / [az] [az] \. youtube \. com http: \ / \ / www \. youtube \. com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# - END - videocache Squid


then save ...

9. create a directory for storing video

indofreebsd # cd / usr/local/www/apache22/data
videocache indofreebsd # mkdir / tmp videocache / youtube videocache / videocache Metacafe / Dailymotion videocache / google videocache / redtube videocache / xtube

videocache / vimeo videocache / wrzuta videocache / YouPorn videocache / Soapbox videocache/tube8 videocache / tvuol videocache / bliptv videocache / break
indofreebsd # chown-R squid: squid videocache


10. Restart all service related

indofreebsd # / usr/local/etc/rc.d/apache22 restart
indofreebsd # / usr / local / etc / rc.d / squid restart


If you successfully install it on videocache log you will like this.

indofreebsd # tail-f / usr / local / squid / logs / videocache.log
32 243 192.168.0.2 2009-05-10 19:28:16,032 - REQUEST - http://v10.lscache7.googlevideo.com/videoplayback?sparams=id% 2Cexpire% 2Cip% 2Cipbits% 2Citag & itag = 34 & ip = 192.168.0.1 & signature = B172DDA7B04663B0F29FBAD60949D5

BAA190CE92.45E154CBB3194D10EF084EA1B63D78A5656080B1 & sver = 3 & expire = 1242006370 & key = yt1 & ipbits = 0 & id = e8add563176c008f
2009-05-10 19:28:16,033 32243 192.168.0.2 e8add563176c008f URL_HIT YOUTUBE

http://v10.lscache7.googlevideo.com/videoplayback?sparams=id% 2Cexpire% 2Cip% 2Cipbits% 2Citag & itag = 34 & ip = 192.168.0.1 & signature = B172DDA7B04663B0F29FBAD60949D5

BAA190CE92.45E154CBB3194D10EF084EA1B63D78A5656080B1 & sver = 3 & expire = 1242006370 & key = yt1 & ipbits = 0 & id = e8add563176c008f
32 243 192.168.0.2 2009-05-10 19:28:16,034 e8add563176c008f CACHE_HIT YOUTUBE Video was served from the cache.
2009-05-10 19:28:16,035 32243 192.168.0.2 e8add563176c008f NEW_URL YOUTUBE

303: http://192.168.0.1/videocache/youtube/e8add563176c008f?sparams=id% 2Cexpire% 2Cip% 2Cipbits% 2Citag & itag = 34 & ip = 192.168.0.1 & signature = B172DDA7B04663B0F2 9FBAD60949D5BAA190CE92.45E154CBB3194D10EF084EA1B63D78A5656080B1 & sver = 3 & expire = 1242006370 & key = yt1 & ipbits = 0 & id = e8add563176c008f


About videocache demikan may be useful.

greetings,
ogeb



Reference:

* Http://cachevideos.com/
* Http://squid-cache.org/
* Http://apache.org

0 comments: