三、简单文件服务器，自动异步缩略图

willvvv

浏览: 328359 次
性别:
来自: 深圳

最近访客更多访客>>

功夫小当家

liuxiao723846

jw1314

yuanyuan7891

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

SimpleFileServer

web.py gearman GraphicsMagick pgmagick thumbnail

1.log.conf，python的日志模块，详见：http://www.red-dove.com/python_logging.html

[loggers]
keys=root,upload,resize

[handlers]
keys=consoleHandler,uploadFileHandler,resizeFileHandler

[formatters]
keys=simpleFormatter

[formatter_simpleFormatter]
format=%(asctime)s %(levelname)s %(message)s

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_upload]
level=DEBUG
handlers=uploadFileHandler
qualname=upload

[logger_resize]
level=DEBUG
handlers=resizeFileHandler
qualname=resize

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[handler_uploadFileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('upload.log',"midnight", 1)

[handler_resizeFileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('resize.log',"midnight", 1)

跟java中的log4j很类似，定义三个logger对象：root,upload,resize，其中root是StreamHandler类型，使用sys.out；upload和resize是TimedRotatingFileHandler，每天生成一个日志文件的方式。

2.conf.py配置文件

FILE_BIG_PATH = '/data/hiluofilev2/big/'
FILE_SMALL_PATH = '/data/hiluofilev2/small/'
HEAD_BIG_PATH = '/data/hiluohead/big/'
HEAD_SMALL_PATH = '/data/hiluohead/small/'

GEARMAN_SERVER = '127.0.0.1:4730'
RESIZE_WORKER = 'resize'
FILE_RESIZE = '140x160!'
HEAD_RESIZE = '86x86!'
IMAGE_FILE = ['jpg','png','gif','jpeg']
AUDIO_FILE = ['amr']

HTTP_FILE = 'http://192.168.0.181:9999/file/'
HTTP_HEAD = 'http://192.168.0.181:9999/head/'

import logging
import logging.config
import os

logging.config.fileConfig(os.path.join(os.path.dirname(os.path.abspath(__file__)), "log.conf"))

FILE_BIG_PATH：这几个PATH表示原图和缩略图保存路径，根据业务区分，用户头像和用户上传的文件分开存放。

GEARMAN_SERVER：gearman Server端的地址。

RESIZE_WORKER：执行图片缩放的worker的名字。

FILE_RESIZE和HEAD_RESIZE：普通图片和头像的缩放尺寸。

IMAGE_FILE和AUDIO_FILE：系统支持的图片格式和音频文件格式，业务上仅支持两种类型。

HTTP_FILE和HTTP_HEAD：文件下载的HTTP前缀。

然后是初始化logging的功能。

3.worker_resize.py 生成缩略图的worker

#!/usr/bin/env python

import os
import gearman
import math
import conf
import logging
import pickle
from pgmagick import Image, FilterTypes

log_resize = logging.getLogger("resize")

class CustomGearmanWorker(gearman.GearmanWorker):  
    def on_job_execute(self, current_job):  
        log_resize.info("Resize Worker started") 
        return super(CustomGearmanWorker, self).on_job_execute(current_job)  
 
def resize_task_callback(gearman_worker, job):  
    filein = job.data.split('-')[0]
    fileout = job.data.split('-')[1]
    fileresize = job.data.split('-')[2]
    log_resize.info('start filein:%s -> fileout:%s' % (filein,fileout))
    im = Image(filein)
    im.quality(80)
    im.filterType(FilterTypes.SincFilter)
    im.scale(fileresize)
    im.sharpen(1.0)
    im.write(fileout) 
    log_resize.info('end filein:%s -> fileout:%s' % (filein,fileout))
    return job.data

new_worker = CustomGearmanWorker([conf.GEARMAN_SERVER])  
new_worker.register_task("resize", resize_task_callback)  
new_worker.work()

jobdata是string，filein代表原图路径，fileout代表缩略图路径，fileresize代表缩放尺寸

4.upload.py 用于上传文件和测试上传文件，使用web.py实现

#!/usr/bin/env python
# coding: utf-8

import web
import conf
import logging
import gearman

urls = (
        '/', 'Index',
        '/upload', 'FileUpload'
       )

app = web.application(urls, globals())
log_upload = logging.getLogger("upload")
# get some directories in conf file
filebigdir = conf.FILE_BIG_PATH 
filesmalldir = conf.FILE_SMALL_PATH 
headbigdir = conf.HEAD_BIG_PATH 
headsmalldir = conf.HEAD_SMALL_PATH 

resize_client = gearman.GearmanClient([conf.GEARMAN_SERVER])

class Index:
    def GET(self):
        web.header("Content-Type","text/html; charset=utf-8")
        return """<html><head></head><body>this hiluofileV2</body></html>"""

class FileUpload:
    def GET(self):
        web.header("Content-Type","text/html; charset=utf-8")
        return """<html><head></head><body>
                <form method="POST" enctype="multipart/form-data" action="">
                username:<input type="text"  name="username" size="20" /><br/>
                md5 or sha1:<input type="text"  name="md5orsha1" size="80"/><br/>
                <input type="file" name="myfile" />
                <br/>
                <input type="submit" />
                </form>
                </body></html>"""

    def POST(self):
        client = web.ctx.environ['REMOTE_ADDR']
        x = web.input(myfile={},username="",md5orsha1="")
        username = x.username.encode('utf-8')
        intusername = 0
        try:
            intusername = int(username)                                             # check if the username is right
        except ValueError:
            log_upload.error('ip:%s username:%s username not right!' % (client,username))
            return 'error[username]'
        md5orsha1 = x.md5orsha1.encode('utf-8')                                     # check if the md5orsha1 is right
        if len(md5orsha1)==32 or len(md5orsha1)==40:
            if 'myfile' in x:                                                       # check if the file-object is created
                filepath = x.myfile.filename.replace('\\','/')
                filename = filepath.split('/')[-1]
                filesuffix = filename.split('.')[-1].lower()
                filename = md5orsha1.lower() + '.' + filesuffix                     # get saved filename(md5orsha1+‘.’+suffix)
                filebig = filebigdir + filename
                if filesuffix in conf.AUDIO_FILE:                                   # audio file just save no need to resize
                    fout = open(filebig,'w')                                        # creates the file where the uploaded file should be stored
                    fout.write(x.myfile.file.read())                                # writes the uploaded file to the newly created file.
                    fout.close()                                                    # closes the file, upload complete.
                    log_upload.info('ip:%s username: %d upload: %s ok!' % (client,intusername,filename))
                    fileurl = conf.HTTP_FILE + 'big/' + filename                    # file download url
                    return fileurl
                elif filesuffix in conf.IMAGE_FILE:
                    filesmall = filesmalldir + filename                             # thumbnail file
                    fileresize = conf.FILE_RESIZE                                   # default FILE_RESIZE
                    fileurl = conf.HTTP_FILE + 'small/' + filename
                    if len(md5orsha1)==40:                                          # user upload head
                        filebig = headbigdir + filename
                        filesmall = headsmalldir + filename                         # thumbnail head
                        fileresize = conf.HEAD_RESIZE                               # thumbnail head HEAD_RESIZE
                        fileurl = conf.HTTP_HEAD + 'small/' + filename
                    fout = open(filebig,'w')                                        # creates the file where the uploaded file should be stored
                    fout.write(x.myfile.file.read())                                # writes the uploaded file to the newly created file.
                    fout.close()                                                    # closes the file, upload complete.
                    log_upload.info('ip:%s username:%d upload:%s ok!' % (client,intusername,filename))
                    jobdata = ('%s-%s-%s' % (filebig,filesmall,fileresize))
                    resize_client.submit_job(conf.RESIZE_WORKER, jobdata)           # submit_job resize and ignore job result
                    log_upload.info('submit resize:%s ok!' % (jobdata))
                    return fileurl
                else:
                    log_upload.error('ip:%s username:%d upload:%s unsupport file type' % (client,intusername,filename))
                    return 'error[filetype]'
            else:
                log_upload.error('ip:%s username:%d upload no myfile' % (client,intusername))
                return 'error[myfile]'
        else:
            log_upload.error('ip:%s username:%d upload:%s unsupport md5orsha1' % (client,intusername,md5orsha1))
            return 'error[filemd5orsha1]'


if __name__ == "__main__":
    #app = web.application(urls, globals())
    web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)
    app.run()

这里业务上定义传头像的key是40位的sha1编码，普通图像文件是32位的md5编码。

5.nginx的nginx.conf修改，使用fastCGI方式运行，

在http节点下增加：

proxy_redirect          off;
proxy_set_header        Host $host;
proxy_set_header        X-Real-IP $remote_addr; #获取真实IP
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffer_size       4k;
proxy_buffers           4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

增加fastCGI静态图片的下载location，nginx开启8099端口代理fastCGI，9999端口用来下载文件

server{
        listen       8099;
        server_name  fast;
        charset      utf-8;

        location / {
            fastcgi_param REQUEST_METHOD $request_method;
            fastcgi_param QUERY_STRING $query_string;
            fastcgi_param CONTENT_TYPE $content_type;
            fastcgi_param CONTENT_LENGTH $content_length;
            fastcgi_param GATEWAY_INTERFACE CGI/1.1;
            fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
            fastcgi_param REMOTE_ADDR $remote_addr;
            fastcgi_param REMOTE_PORT $remote_port;
            fastcgi_param SERVER_ADDR $server_addr;
            fastcgi_param SERVER_PORT $server_port;
            fastcgi_param SERVER_NAME $server_name;
            fastcgi_param SERVER_PROTOCOL $server_protocol;
            fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_script_name;
            fastcgi_pass 127.0.0.1:8091;
        }
    }

    server
    {
        listen       9999;
        server_name  file.hiluo.cn;

        location /file/ {
            alias  /data/hiluofilev2/;
        }    

        location /head/ {
            alias  /data/hiluohead/;
        }    
     }

注意，在配置nginx的fastcgi时可能会报错如下：

5.1.child exited with 2
解决方法: insert #!/usr/bin/env python into header of main.py
5.2.spawn-fcgi child exited with 126
解决方法: chmod +x upload.py

5.3.child exited with1

这时可以先修改upload.py，屏蔽web.wsgi.runwsgi行，打开app = web.application(urls, globals())行，用python upload.py 8091方式启动测试。

6.hiluo-file-http.sh，runwsgi的启动停止管理脚本

#!/bin/sh

# 
# hiluo-file-http control
# $Date: 2012-07-04 willzhai $
#

# If pid file path is not set elsewhere, set to /tmp/hiluo-file-http.pid
[ -z "$PIDFILE" ] && PIDFILE="/tmp/hiluo-file-http.pid"

dir=/root/test-file-v2

start() {
    echo "Starting hiluo-file-http..."
    exec_result=`spawn-fcgi -d $dir -f $dir/upload.py -a 127.0.0.1 -p 8091`
    echo $exec_result
    if [ ! -z "$PIDFILE" ]; then
        echo $exec_result|awk -F ':' '{print $4}' > $PIDFILE
    fi
}

stop() {
    # Stop daemons.
    echo "Shutting down hiluo-file-http..."
    kill `pgrep -f "python $dir/upload.py"`
    rm -f $PIDFILE
}

restart() {
    stop
    sleep 3 # give it a few moments to shut down
    start
}

status() {
    pid=`cat $PIDFILE 2>&1`
    if [ "$?" = "1" ]; then
        echo "hiluo-file-http is not running"
        RETVAL=0
    else 
        ps -p $pid > /dev/null 2>&1
        if [ "$?" = "0" ]; then 
            echo "hiluo-file-http is running"
            RETVAL=0
        else 
            echo "hiluo-file-http is not running"
            RETVAL=0
        fi
    fi
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        restart
        ;;
    status) 
        status
        ;;
    *)
        echo "Usage $0 {start|stop|restart|status}"
        RETVAL=1
esac

exit $?

注意将dir改成实际的部署路径。这个可以再改进，自动识别当前路径。

7.hiluo-file-resize.sh ，resize worker的启动停止管理脚本

#!/bin/sh

# 
# hiluo-file-resize control
# $Date: 2012-07-04 willzhai $
#

# If pid file path is not set elsewhere, set to /tmp/hiluo-file-resize.pid
[ -z "$PIDFILE" ] && PIDFILE="/tmp/hiluo-file-resize.pid"

dir=/root/test-file-v2

start() {
    echo "Starting hiluo-file-resize..."
    exec_command="exec python $dir/worker_resize.py 2>&1 &"
    eval $exec_command
    RETVAL=$?

    if [ $RETVAL -eq 0 -a ! -z "$PIDFILE" ]; then
        echo $! > $PIDFILE
    fi

    echo "hiluo-file-resize new pid: "`cat $PIDFILE`
}

stop() {
    # Stop daemons.
    echo "Shutting down hiluo-file-resize..."
    kill `pgrep -f "python $dir/worker_resize.py"`
    rm -f $PIDFILE
}

restart() {
    stop
    sleep 3 # give it a few moments to shut down
    start
}

status() {
    pid=`cat $PIDFILE 2>&1`
    if [ "$?" = "1" ]; then
        echo "hiluo-file-resize is not running"
        RETVAL=0
    else 
        ps -p $pid > /dev/null 2>&1
        if [ "$?" = "0" ]; then 
            echo "hiluo-file-resize is running"
            RETVAL=0
        else 
            echo "hiluo-file-resize is not running"
            RETVAL=0
        fi
    fi
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        restart
        ;;
    status) 
        status
        ;;
    *)
        echo "Usage $0 {start|stop|restart|status}"
        RETVAL=1
esac

exit $?

注意将dir改成实际的部署路径。这个可以再改进，自动识别当前路径。

8.效果如下

8.1上传页面