ceph对象存储折腾记

2019独角兽企业重金招聘Python工程师标准>>>

前言

一直想弄对象存储，以前弄过一次，不是很理解region是个什么东西，后来时间和工作上的原因没有再折腾，这两天闲了下来，再次折腾了一次。我是参考的ceph的中文翻译文挡进行的部署和测试。传送门,文档里面介绍的和ceph本身的版本存在脱节的现象，可能初次接触的人会因为服务启动的问题摸不着头脑。 本文档只适用于ceph jewel版本的部署

关于部署

安装ceph必要的软件包，配置好公共密钥和ceph mon的配置，这里我不再谈了。 对象存储额外需要安装的包是：ceph-radosgw和ceph-common 安装完毕你的系统上应该至少存在三个命令：rados 、 radosgw 、 radosgw-admin 其中整个对象网关服务就是由radosgw来启动的，radosgw-admin负责管理对象资源（用户，权限，bucket），rados基本算一个比较简单的s3客户端（？我这里可能理解不是很精确）

配置

ceph.conf

[global] fsid = xxxxxxxxxxxxxxxxxxxxxxxxxxxx mon_initial_members = t41,t42,t45 mon_host = 192.168.168.41,192.168.168.42,192.168.168.45 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx# t56是服务器的hostname,由hostname -s命令可获取 [client.radosgw.t56] host = your_ceph_rados_host keyring = /etc/ceph/ceph.client.radosgw.keyring rgw_socket_path = "/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock" log_file = "/data/logs/client.radosgw.gateway.log" rgw_frontends = civetweb port=80 rgw_print_continue = true

所有配置选项参考传送门

密钥环

#创建方式 ceph auth create client.radosgw.t56 osd 'allow rwx' mon 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring #密钥环的样子 [client.radosgw.t56]key = xxxxxxxxxxxxxxxxxxxxxxxx==

服务的启动方式

#有三种启动方式 1. /etc/init.d/ceph-radosgw [start|stop|status|reload] 2. systemctl start ceph-radosgw 3. radosgw -c /etc/ceph/ceph.conf -n client.radosgw.t56

本质都是第三种启动，无非写了个脚本而已。radosgw -h可以看看其他的参数，其中-f前台执行和--debug_ms设置调试等级有利于调试。

授权用户

[root@t56 /data]# radosgw-admin user create --uid="xueyi28" --display-name="Xueyi" {"user_id": "xueyi28","display_name": "Xueyi","email": "","suspended": 0,"max_buckets": 1000,"auid": 0,"subusers": [],"keys": [{"user": "xueyi28","access_key": "xxxxxxxxxxxxxxx","secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}],"swift_keys": [],"caps": [],"op_mask": "read, write, delete","default_placement": "","placement_tags": [],"bucket_quota": {"enabled": false,"max_size_kb": -1,"max_objects": -1},"user_quota": {"enabled": false,"max_size_kb": -1,"max_objects": -1},"temp_url_keys": [] }

• 这里有一个要注意的，这数据格式是json格式，access_key和secret_key上可能存在反斜杠之类的转意字符，用key的时候，注意把转意字符处理一下，省的纠结生成的key老是验证不过。

关于region

对象存储设计考虑到数据中心区域的问题，这里的region就是区域的标识。比如中国西部数据中心，中国南方数据中心，不同的区域数据中心的bucket还可以异地同步（？下一步需要研究的），配合用户的DNS，可以让不同区域的用户连接不同区域的数据中心数据。

[root@t56 /data]# radosgw-admin region get {"name": "default","api_name": "","is_master": "true","endpoints": [],"hostnames": [],"master_zone": "","zones": [{"name": "default","endpoints": [],"log_meta": "false","log_data": "false","bucket_index_max_shards": 0}],"placement_targets": [{"name": "default-placement","tags": []}],"default_placement": "default-placement" }[root@t56 /data]# radosgw-admin regions list {"default_info": {"default_region": "default"},"regions": ["default"] }

我这个是测试用的，就只有一个默认的default region。

数据的读写和bucket的使用

我这里主要说php sdk使用s3接口。ceph给的文档里面的大多数sdk版本都是上个世纪的，亚马逊的s3 php sdk变的乱七八糟，入门比较慢，搞半天也搞不明白。我留了一个老版本的php sdk，凑合能接到ceph的文档。 测试代码

<\?php define('AWS_KEY', 'your_access_key'); define('AWS_SECRET_KEY', 'your_secret_key'); define('AWS_CANONICAL_ID', 'xueyi28'); define('AWS_CANONICAL_NAME', 'Xueyi'); $HOST = 'your_ceph_radosgw_host';// require the amazon sdk for php library require_once 'AWSSDKforPHP/sdk.class.php';// Instantiate the S3 class and point it at the desired host $Connection = new AmazonS3(array('key' => AWS_KEY,'secret' => AWS_SECRET_KEY,'canonical_id' => AWS_CANONICAL_ID,'canonical_name' => AWS_CANONICAL_NAME, )); $Connection->use_ssl = false; //禁用ssl $Connection->set_hostname($HOST); $Connection->enable_path_style(true); //采用path的模式，不然就是域名模式，bucket会成为根域名的子域名 $Connection->allow_hostname_override(false); $Connection->path_style = true; //sdk哪里有点问题，path_style配置老是不生效，可以调一下sdk的代码 #$Connection->create_bucket('my-new-bucket', AmazonS3::REGION_US_E1); //region可以在代码中进行定制，一个region对应一个地方域名。我们是default region 就对应REGION_US_E1，默认的就这个，看代码就明白了 $ListResponse = $Connection->list_buckets(); $Buckets = $ListResponse->body->Buckets->Bucket; foreach ($Buckets as $Bucket) {echo $Bucket->Name . "\t" . $Bucket->CreationDate . "\n"; } $Connection->create_object('my-new-bucket', 'hello.txt', array('body' => "Hello World!", )); $Connection->set_object_acl('my-new-bucket', 'hello.txt', AmazonS3::ACL_PUBLIC);

php sdk传送门 核心逻辑代码/services/s3.class.php

终端玩转对象存储

在终端下要用s3对象存储，最好的工具无非是s3cmd,一般yum就可以直接安装下来，不过这玩意要注意版本，老版本的和最新的文档用法差别比较大。

首先要定制一下s3cmd的配置

[root@t56 ~]# s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options.Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: xxxxxxxxxxxxxxxxxxxxxxxx Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Default Region [US]: defaultEncryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [No]: NoOn some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: New settings:Access Key: xxxxxxxxxxxxxxxxxxxxxxxxxxSecret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxDefault Region: defaultEncryption password: Path to GPG program: /usr/bin/gpgUse HTTPS protocol: FalseHTTP Proxy server name: HTTP Proxy server port: 0Test access with supplied credentials? [Y/n] nSave settings? [y/N] y Configuration saved to '/root/.s3cfg'

定制完之后要再编辑一下/root/.s3cfg文件，把你自定义的Host写进去，定义好你的Host和region、bucket的访问方式 给一个例子

[root@t56 ~]# cat .s3cfg [default] access_key = xxxxxxxxxxxxxxxxxxxxxxx access_token = add_encoding_exts = add_headers = bucket_location = default ca_certs_file = cache_file = check_ssl_certificate = True cloudfront_host = your_ceph_rgw_host default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = ANSI_X3.4-1968 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = your_ceph_rgw_host host_bucket = your_ceph_rgw_host/%(bucket) human_readable_sizes = False ignore_failed_copy = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False list_md5 = False log_target_prefix = max_delete = -1 mime_type = multipart_chunk_size_mb = 15 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 put_continue = False recursive = False recv_chunk = 4096 reduced_redundancy = False restore_days = 1 secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx send_chunk = 4096 server_side_encryption = False signature_v2 = False simpledb_host = sdb.your_ceph_rgw_host skip_existing = False socket_timeout = 300 urlencoding_mode = normal use_https = False use_mime_magic = True verbosity = WARNING website_endpoint = http://%(bucket)s.s3-website-%(location)your_ceph_rgw_host/ website_error = website_index = index.html

有几个地方修改你得按照你自己的需求去整，需要什么就整什么，让我们试试

[root@t56 ~]# s3cmd ls 2016-11-23 12:36 s3://my-new-bucket [root@t56 ~]# s3cmd ls s3://my-new-bucket 2016-11-23 12:38 12 s3://my-new-bucket/hello.txt [root@t56 ~]# s3cmd put test.log s3://my-new-bucket/test.log test.log -> s3://my-new-bucket/test.log [1 of 1]242 of 242 100% in 0s 3.48 kB/s done [root@t56 ~]# s3cmd setacl s3://my-new-bucket/test.log --acl-public s3://my-new-bucket/test.log: ACL set to Public [1 of 1] [root@t56 ~]# s3cmd del s3://my-new-bucket/hello.txt File s3://my-new-bucket/hello.txt deleted//递归授权的方式 s3cmd setacl s3://myexamplebucket.calvium.com/ --acl-public --recursive

棒极了

转载于:https://my.oschina.net/xueyi28/blog/793600

总结

以上是生活随笔为你收集整理的ceph对象存储折腾记的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。