听云 Server for Python 探针公测开启，现在加入，机械键盘抱回家！

[TINGYUN]

24h 紧盯显示器的运维为何忽然神秘地有了空余时间？
数年如一日做好救险准备的他们为何突然能正常下班吃上生蚝撸大腰？
改变运维的命运，究竟是何人所为？运维工作变得轻松的背后又隐藏着什么？
这一切的一切，
是工作倦怠开始偷懒？还是准备跳槽另有隐情？
是捡到秘籍修得神功？还是打通二脉融会贯通？
尽请关注《走进科学》暑期特别节目《当运维偶遇听云》，让我们跟随着镜头走进这不为人知的秘密...

听云 Server

---

• 分钟级定位代码，数据库， NoSQL ，后端 API 接口，第三方服务造成的性能问题和故障，给出细致的性能问题解读
• 独有的对慢 SQL 追踪功能，通过对 SQL 语句、执行计划和代码调用堆栈的分析，直接发现慢 SQL 的病因
• 支持对关系型数据库服务，及 Redis, Memcached, MongoBD 等新兴 NoSQL 服务进行性能分析
• 针对 500 余种服务端应用常见错误，提供详细到代码行的追踪数据，终结 bug -3 分钟内收到报警模块提醒，在用户投诉前掌握先机

领奖姿势

---

听云身为 SaaS 领域唯一上市新三板的公司, 对加入公测的同学的奖励，也大方的不要不要的:

成功部署听云 Server for Python 探针，发送截图及账户名至 [email protected] ， 前 50 名部署成功同学（以收到邮件时间为准）, 雷蛇游戏键鼠套装为您双手奉上！

那么问题来了，如何加入公测呢

---

1. 来，少年，请先申请一个听云账号 注册来戳我呀 ， 并开通听云 Server
2. 下载并安装 Python 探针
3. 重启或重新启动应用服务器
4. 成功部署 Python 探针，监控服务端应用性能，从此老板再也不用担心我因问题发现太晚而造成用户流失，开启人生 easy 模式！

活动细则

---

• 公测周期 3 个月，期间 Python 探针 beta 版本的功能和应用数都不受限制
• 发送部署成功的截图需包含 Python 字样及应用名称等关键信息，以便于工作人员核实
• 核实部署无误后，会得到邮件确认，奖品会在活动结束后统一发放，请表催我> <
• 听云 Server 技术交流群： QQ 332097173 ，欢迎加入:D
• 关于本活动若有其他疑问，请尽情邮件 [email protected]
• 同学们踊跃在帖子里发言呐

每款应用都需要听云

[glasslion]

连 pypi 都不上传一个， 不觉得很 low 吗

[zjxubinbin]

好,我来试试先~

[zjxubinbin]

部署了半天没数据,发现竟然只支持 Django,看来我等用 Flask 的与键盘无缘了,哎~我还是洗洗睡吧~

[defunct9]

为了抢键盘，居然用手机注册账号，上服务器装了 Sys 探针发了邮件。回来发现居然要 python 探针，装上用 flask,失败。再来， django,成功。拼了！

[le0rn0]

渣渣，大晚上的给反馈个 bug 都没人理

[lins05]

license key 旁边的复制按钮的英文应该用 copy 而不是 duplicate 吧

[glasslion]

Copyright (c ) 2010-2015 New Relic, Inc. All rights reserved.

Certain inventions disclosed in this file may be claimed within
patents owned or patent applications filed by New Relic, Inc. or third
parties.

Subject to the terms of this notice, New Relic grants you a
nonexclusive, nontransferable license, without the right to
sublicense, to (a ) install and execute one copy of these files on any
number of workstations owned or controlled by you and (b ) distribute
verbatim copies of these files to third parties. As a condition to the
foregoing grant, you must provide this notice along with each copy you
distribute and you must not remove, alter, or obscure this notice. All
other use, reproduction, modification, distribution, or other
exploitation of these files is strictly prohibited, except as may be set
forth in a separate written license agreement between you and New
Relic. The terms of any such license agreement will control over this
notice. The license stated above will be automatically terminated and
revoked if you exceed its scope or violate any of the terms of this
notice.

This License does not grant permission to use the trade names,
trademarks, service marks, or product names of New Relic, except as
required for reasonable and customary use in describing the origin of
this file and reproducing the content of this notice. You may not
mark or brand this file with any trade name, trademarks, service
marks, or product names other than the original brand (if any )
provided by New Relic.

Unless otherwise expressly agreed by New Relic in a separate written
license agreement, these files are provided AS IS, WITHOUT WARRANTY OF
ANY KIND, including without any implied warranties of MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE, or NON-INFRINGEMENT. As a
condition to your use of these files, you are solely responsible for
such use. New Relic will have no liability to you for direct,
indirect, consequential, incidental, special, or punitive damages or
for lost profits or data.

[glasslion]

@Livid 这个产品的 Python 探针 代码是 完全抄袭竞争对手 New Relic 的 https://pypi.python.org/pypi/newrelic

[glasslion]

操， 公平起见， 顺便揭露下， 你们的国内的竞争对手 OneAPM 也是抄 New Relic 的

https://www.v2ex.com/t/125736

[Strikeactor]

@glasslion 隐隐感觉水深啊

前排看戏，瓜子鸡脚矿泉水

[beordle]

国内的常态了，但如果对源码改动增强很多的话，我是可以接受的。

[glasslion]

@beordle 几乎没有改动

[zjxubinbin]

@glasslion 莫不是又一个骗投资人的?

[est]

@glasslion
@Strikeactor
@beordle

copy to chinar 。

[alsotang]

newrelic 到底应该最更强的混淆。。还是应该直接开源呢。。。

[TINGYUN]

@glasslion https://pypi.python.org/pypi/tingyun 这里这里
非常感谢关注听云，欢迎提出更多宝贵的意见。

[TINGYUN]

@zjxubinbin 听云的 Python 探针是在开源的基础上、公司内部数据协议上自行研发的，对于一些其他框架、插件的开发，将会逐渐的支持。
另据研发同学的可靠消息，对 Tornado ， flask 的支持，很快就会实现啦。

[TINGYUN]

@lins05 谢谢反馈，已改成 copy

[lins05]

@TINGYUN :+1:

[TINGYUN]

@glasslion 同学提抄袭这词有些严肃啊，那就认真的回复下这个质疑（明明上面的回复也很认真）@Strikeactor @beordle @zjxubinbin @alsotang
首先，听云绝对不会抄袭的。
听云和 New Relic 都是基于第三方的开源模块 wrapt ，加自主研发。两家都大量借用了 wrapt, requests 模块相关功能。某一语言的探针的监测原理是一样的，基于开源组件，部分源码有重叠，是自然的事情。

下面是三段听云和 New Relic 核心代码的对照。

tingyun-hooks_entrance.py
//i.v2ex.co/d0c12Dn1.png
//i.v2ex.co/TVII4A7b.png

newrelic-weib_transaction.py
import sys
import cgi
import base64
import time
import string
import re
import json
import logging

try:
import urlparse
except ImportError:
import urllib.parse as urlparse

import newrelic.packages.six as six

import newrelic.api.application
import newrelic.api.transaction
import newrelic.api.object_wrapper
import newrelic.api.function_trace
_logger = logging.getLogger (__name__)

_rum_header_fragment = '<script type="text/javascript">' \
'var NREUMQ=NREUMQ||[];NREUMQ.push (["mark","firstbyte",' \
'new Date ().getTime ()]);</script>'

_rum_footer_short_fragment = '<script type="text/javascript">' \
'if (!NREUMQ.f ){NREUMQ.f=function (){NREUMQ.push (["load",' \
'new Date ().getTime ()]);if (NREUMQ.a )NREUMQ.a ();};' \
'NREUMQ.a=window.onload;window.onload=NREUMQ.f;};' \
'NREUMQ.push (["nrf2","%s","%s","%s","%s",%d,%d,' \
'new Date ().getTime ()]);</script>'

_rum2_footer_short_fragment = '<script type="text/javascript">' \
'if (!NREUMQ.f ){NREUMQ.f=function (){NREUMQ.push (["load",' \
'new Date ().getTime ()]);if (NREUMQ.a )NREUMQ.a ();};' \
'NREUMQ.a=window.onload;window.onload=NREUMQ.f;};' \
'NREUMQ.push (["nrfj","%s","%s","%s","%s",%d,%d,' \
'new Date ().getTime (),"%s","%s","%s","%s","%s"]);</script>'

_rum_footer_long_fragment = '<script type="text/javascript">' \
'if (!NREUMQ.f ){NREUMQ.f=function (){NREUMQ.push (["load",' \
'new Date ().getTime ()]);var e=document.createElement ("script");' \
'e.type="text/javascript";' \
'e.src=(("http:"===document.location.protocol )?"http:":"https:")' \
'+"//"+"%s";document.body.appendChild (e );if (NREUMQ.a )NREUMQ.a ();};' \
'NREUMQ.a=window.onload;window.onload=NREUMQ.f;};' \
'NREUMQ.push (["nrf2","%s","%s","%s","%s",%d,%d,' \
'new Date ().getTime ()]);</script>'

_rum2_footer_long_fragment = '<script type="text/javascript">' \
'if (!NREUMQ.f ){NREUMQ.f=function (){NREUMQ.push (["load",' \
'new Date ().getTime ()]);var e=document.createElement ("script");' \
'e.type="text/javascript";' \
'e.src=(("http:"===document.location.protocol )?"http:":"https:")' \
'+"//"+"%s";document.body.appendChild (e );if (NREUMQ.a )NREUMQ.a ();};' \
'NREUMQ.a=window.onload;window.onload=NREUMQ.f;};' \
'NREUMQ.push (["nrfj","%s","%s","%s","%s",%d,%d,' \
'new Date ().getTime (),"%s","%s","%s","%s","%s"]);</script>'

# Seconds since epoch for Jan 1 2000

JAN_1_2000 = time.mktime ((2000, 1, 1, 0, 0, 0, 0, 0, 0 ))

def _encode (name, key ):
s = []

# Convert name and key into bytes which are treated as integers.

key = list (six.iterbytes (six.b (key )))
for i, c in enumerate (six.iterbytes (six.b (name ))):
s.append (chr (c ^ key[i % len (key )]))
return s

if six.PY3:
def obfuscate (name, key ):
if not (name and key ):
return ''

# Always pass name and key as str to _encode ()

return str (base64.b64encode (six.b (''.join (_encode (name, key )))),
encoding='Latin-1')
else:
def obfuscate (name, key ):
if not (name and key ):
return ''

# Always pass name and key as str to _encode ()

return base64.b64encode (six.b (''.join (_encode (name, key ))))

def deobfuscate (name, key ):
if not (name and key ):
return ''

# Always pass name and key as str to _encode ()

return ''.join (_encode (six.text_type (base64.b64decode (six.b (name )),
encoding='Latin-1'), key ))

def _lookup_environ_setting (environ, name, default=False ):
flag = environ.get (name, default )
if default is None or default:
try:
flag = not flag.lower () in ['off', 'false', '0']
except AttributeError:
pass
else:
try:
flag = flag.lower () in ['on', 'true', '1']
except AttributeError:
pass
return flag

def _extract_token (cookie ):
try:
t = re.search (r"\bNRAGENT=(tk=.{16})", cookie )
token = re.search (r"^tk=([^\"<'>]+)$", t.group (1 )) if t else None
return token and token.group (1 )
except Exception:
pass


class WebTransaction (newrelic.api.transaction.Transaction ):

def __init__(self, application, environ ):

# The web transaction can be enabled/disabled by
# the value of the variable "newrelic.enabled"
# in the WSGI environ dictionary. We need to check
# this before initialising the transaction as needs
# to be passed in base class constructor. The
# default is None, which would then result in the
# base class making the decision based on whether
# application or agent as a whole are enabled.

enabled = _lookup_environ_setting (environ, 'newrelic.enabled', None )

# Initialise the common transaction base class.

newrelic.api.transaction.Transaction.__init__(self, application, enabled )

# Bail out if the transaction is running in a
# disabled state.

if not self.enabled:
return

# Will need to check the settings a number of times.

settings = self._settings

# Check for override settings from WSGI environ.

self.background_task = _lookup_environ_setting (environ, 'newrelic.set_background_task', False )

self.ignore_transaction = _lookup_environ_setting (environ,
'newrelic.ignore_transaction', False )
self.suppress_apdex = _lookup_environ_setting (environ,
'newrelic.suppress_apdex_metric', False )
self.suppress_transaction_trace = _lookup_environ_setting (environ,
'newrelic.suppress_transaction_trace', False )
self.capture_params = _lookup_environ_setting (environ,
'newrelic.capture_request_params',
settings.capture_params )
self.autorum_disabled = _lookup_environ_setting (environ,
'newrelic.disable_browser_autorum',
not settings.browser_monitoring.auto_instrument )

# Extract from the WSGI environ dictionary
# details of the URL path. This will be set as
# default path for the web transaction. This can
# be overridden by framework to be more specific
# to avoid metrics explosion problem resulting
# from too many distinct URLs for same resource
# due to use of REST style URL concepts or
# otherwise.

request_uri = environ.get ('REQUEST_URI', None )
script_name = environ.get ('SCRIPT_NAME', None )
path_info = environ.get ('PATH_INFO', None )
http_cookie = environ.get ('HTTP_COOKIE', None )

if http_cookie and ("NRAGENT" in http_cookie ):
self.rum_token = _extract_token ( http_cookie )
self.rum_trace = True if self.rum_token else False

self._request_uri = request_uri

if self._request_uri is not None:
# Need to make sure we drop off any query string
# arguments on the path if we have to fallback
# to using the original REQUEST_URI. Can't use
# attribute access on result as only support for
# Python 2.5+.

self._request_uri = urlparse.urlparse (self._request_uri )[2]

if script_name is not None or path_info is not None:
if path_info is None:
path = script_name
elif script_name is None:
path = path_info
else:
path = script_name + path_info

self.set_transaction_name (path, 'Uri', priority=1 )

if self._request_uri is None:
self._request_uri = path
else:
if self._request_uri is not None:
self.set_transaction_name (self._request_uri, 'Uri', priority=1 )

# See if the WSGI environ dictionary includes the
# special 'X-Request-Start' or 'X-Queue-Start' HTTP
# headers. These header are optional headers that can be
# set within the underlying web server or WSGI server to
# indicate when the current request was first received
# and ready to be processed. The difference between this
# time and when application starts processing the
# request is the queue time and represents how long
# spent in any explicit request queuing system, or how
# long waiting in connecting state against listener
# sockets where request needs to be proxied between any
# processes within the application server.
#
# Note that mod_wsgi sets its own distinct variables
# automatically. Initially it set mod_wsgi.queue_start,
# which equated to when Apache first accepted the
# request. This got changed to mod_wsgi.request_start
# however, and mod_wsgi.queue_start was instead used

[TINGYUN]

@glasslion
tingyun-memcache_trace.py

import logging

from tingyun.api.tracert.memcache_node import MemcacheNode
from tingyun.api.objects.object_wrapper import wrap_object, FunctionWrapper
from tingyun.api.tracert.time_trace import TimeTrace
from tingyun.api.transaction.base import current_transaction

_logger = logging.getLogger (__name__)


class MemcacheTrace (TimeTrace ):
def __init__(self, transaction, command ):
super (MemcacheTrace, self ).__init__(transaction )
self.command = command

def create_node (self ):
return MemcacheNode (command=self.command, children=self.children, start_time=self.start_time,
end_time=self.end_time, duration=self.duration, exclusive=self.exclusive )

def terminal_node (self ):
return True


def memcached_trace_wrapper (wrapped, command ):
"""
:return:
"""
def dynamic_wrapper (wrapped, instance, args, kwargs ):
transaction = current_transaction ()
if transaction is None:
return wrapped (*args, **kwargs )

if instance is not None:
_command = command (instance, *args, **kwargs )
else:
_command = command (*args, **kwargs )

with MemcacheTrace (transaction, _command ):
return wrapped (*args, **kwargs )

def literal_wrapper (wrapped, instance, args, kwargs ):
transaction = current_transaction ()
if transaction is None:
return wrapped (*args, **kwargs )

with MemcacheTrace (transaction, command ):
return wrapped (*args, **kwargs )

if callable (command ):
return FunctionWrapper (wrapped, dynamic_wrapper )

return FunctionWrapper (wrapped, literal_wrapper )


# egg: (memcached, Client.append, get )
def wrap_memcache_trace (module, object_path, command ):
wrap_object (module, object_path, memcached_trace_wrapper, (command,))

newrelic-memcache_trace.py

import sys
import types
import time
import logging

import newrelic.core.memcache_node

import newrelic.api.transaction
import newrelic.api.time_trace
import newrelic.api.object_wrapper


_logger = logging.getLogger (__name__)


class MemcacheTrace (newrelic.api.time_trace.TimeTrace ):

node = newrelic.core.memcache_node.MemcacheNode

def __init__(self, transaction, command ):
super (MemcacheTrace, self ).__init__(transaction )

self.command = command

def dump (self, file ):
print >> file, self.__class__.__name__, dict (command=self.command )

def create_node (self ):
return self.node (command=self.command, children=self.children,
start_time=self.start_time, end_time=self.end_time,
duration=self.duration, exclusive=self.exclusive )

def terminal_node (self ):
return True

class MemcacheTraceWrapper (object ):

def __init__(self, wrapped, command ):
if isinstance (wrapped, tuple ):
(instance, wrapped ) = wrapped
else:
instance = None

newrelic.api.object_wrapper.update_wrapper (self, wrapped )

self._nr_instance = instance
self._nr_next_object = wrapped

if not hasattr (self, '_nr_last_object'):
self._nr_last_object = wrapped

self._nr_command = command

def __get__(self, instance, klass ):
if instance is None:
return self
descriptor = self._nr_next_object.__get__(instance, klass )
return self.__class__((instance, descriptor ), self._nr_command )

def __call__(self, *args, **kwargs ):
transaction = newrelic.api.transaction.current_transaction ()

if not transaction:
return self._nr_next_object (*args, **kwargs )

if callable (self._nr_command ):
if self._nr_instance is not None:
command = self._nr_command (self._nr_instance, *args,
**kwargs )
else:
command = self._nr_command (*args, **kwargs )
else:
command = self._nr_command

with MemcacheTrace (transaction, command ):
return self._nr_next_object (*args, **kwargs )

def memcache_trace (command ):
def decorator (wrapped ):
return MemcacheTraceWrapper (wrapped, command )
return decorator

def wrap_memcache_trace (module, object_path, command ):
newrelic.api.object_wrapper.wrap_object (module, object_path,
MemcacheTraceWrapper, (command,))

[TINGYUN]

@glasslion
tingyun-database.redis.py
from tingyun.api.tracert.redis_trace import wrap_redis_trace
from tingyun.api.tracert.function_trace import wrap_function_trace

rewrite_command = ['setex', 'lrem', 'zadd']
basic_command = [
'append', 'bgrewriteaof', 'bgsave', 'bitcount', 'bitop', 'bitpos', 'blpop', 'brpop', 'brpoplpush', 'client_getname',
'client_kill', 'client_list', 'client_setname', 'config_get', 'config_resetstat', 'config_rewrite', 'config_set',
'connection_pool', 'dbsize', 'debug_object', 'decr', 'delete', 'dump', 'echo', 'eval', 'evalsha', 'execute_command',
'exists', 'expire', 'expireat', 'flushall', 'flushdb', 'from_url', 'get', 'getbit', 'getrange', 'getset', 'hdel',
'hexists', 'hget', 'hgetall', 'hincrby', 'hincrbyfloat', 'hkeys', 'hlen', 'hmget', 'hmset', 'hscan', 'hscan_iter',
'hset', 'hsetnx', 'hvals', 'incr', 'incrby', 'incrbyfloat', 'info', 'keys', 'lastsave', 'lindex', 'linsert', 'llen',
'lock', 'lpop', 'lpush', 'lpushx', 'lrange', 'lrem', 'lset', 'ltrim', 'mget', 'move', 'mset', 'msetnx',
'object', 'parse_response', 'persist', 'pexpire', 'pexpireat', 'pfadd', 'pfcount', 'pfmerge', 'ping',
'pipeline', 'psetex', 'pttl', 'publish', 'pubsub', 'randomkey', 'register_script', 'rename',
'renamenx', 'response_callbacks', 'restore', 'rpop', 'rpoplpush', 'rpush', 'rpushx', 'sadd', 'save',
'scan', 'scan_iter', 'scard', 'script_exists', 'script_flush', 'script_kill', 'script_load', 'sdiff',
'sdiffstore', 'set', 'set_response_callback', 'setbit', 'setex', 'setnx', 'setrange', 'shutdown',
'sinter', 'sinterstore', 'sismember', 'slaveof', 'slowlog_get', 'slowlog_len', 'slowlog_reset',
'smembers', 'smove', 'sort', 'spop', 'srandmember', 'srem', 'sscan', 'sscan_iter', 'strlen', 'substr',
'sunion', 'sunionstore', 'time', 'transaction', 'ttl', 'type', 'unwatch', 'watch', 'zadd', 'zcard',
'zcount', 'zincrby', 'zinterstore', 'zlexcount', 'zrange', 'zrangebylex', 'zrangebyscore', 'zrank',
'zrem', 'zremrangebylex', 'zremrangebyrank', 'zremrangebyscore', 'zrevrange', 'zrevrangebyscore',
'zrevrank', 'zscan', 'zscan_iter', 'zscore', 'zunionstore']


def detect_connection (module ):
"""
:param module:
:return:
"""
if hasattr (module, 'Connection') and hasattr (module.Connection, 'connect'):
wrap_function_trace (module, "Connection.connect", name="connect")


def detect_client_operation (module ):
"""
:param module:
:return:
"""
if hasattr (module, 'StrictRedis'):
for command in basic_command:
if hasattr (module.StrictRedis, command ):
wrap_redis_trace (module, "StrictRedis.%s" % command, command )

if hasattr (module, 'Redis'):
for command in rewrite_command:
if hasattr (module.Redis, command ):
wrap_redis_trace (module, "Redis.%s" % command, command )
elif hasattr (module, 'Redis'):
for command in basic_command:
if hasattr (module.Redis, command ):
wrap_redis_trace (module, "Redis.%s" % command, command )

newrelic_database.nosql_redis.py
import newrelic.api.function_trace

_methods_1 = ['bgrewriteaof', 'bgsave', 'config_get', 'config_set',
'dbsize', 'debug_object', 'delete', 'echo', 'flushall',
'flushdb', 'info', 'lastsave', 'object', 'ping', 'save',
'shutdown', 'slaveof', 'append', 'decr', 'exists',
'expire', 'expireat', 'get', 'getbit', 'getset', 'incr',
'keys', 'mget', 'mset', 'msetnx', 'move', 'persist',
'randomkey', 'rename', 'renamenx', 'set', 'setbit',
'setex', 'setnx', 'setrange', 'strlen', 'substr', 'ttl',
'type', 'blpop', 'brpop', 'brpoplpush', 'lindex',
'linsert', 'llen', 'lpop', 'lpush', 'lpushx', 'lrange',
'lrem', 'lset', 'ltrim', 'rpop', 'rpoplpush', 'rpush',
'rpushx', 'sort', 'sadd', 'scard', 'sdiff', 'sdiffstore',
'sinter', 'sinterstore', 'sismember', 'smembers',
'smove', 'spop', 'srandmember', 'srem', 'sunion',
'sunionstore', 'zadd', 'zcard', 'zcount', 'zincrby',
'zinterstore', 'zrange', 'zrangebyscore', 'zrank', 'zrem',
'zremrangebyrank', 'zremrangebyscore', 'zrevrange',
'zrevrangebyscore', 'zrevrank', 'zscore', 'zunionstore',
'hdel', 'hexists', 'hget', 'hgetall', 'hincrby', 'hkeys',
'hlen', 'hset', 'hsetnx', 'hmset', 'hmget', 'hvals',
'publish']

_methods_2 = ['setex', 'lrem', 'zadd']

def instrument_redis_connection (module ):

newrelic.api.function_trace.wrap_function_trace (
module, 'Connection.connect')

def instrument_redis_client (module ):

if hasattr (module, 'StrictRedis'):
for method in _methods_1:
if hasattr (module.StrictRedis, method ):
newrelic.api.function_trace.wrap_function_trace (
module, 'StrictRedis.%s' % method )
else:
for method in _methods_1:
if hasattr (module.Redis, method ):
newrelic.api.function_trace.wrap_function_trace (
module, 'Redis.%s' % method )

for method in _methods_2:
if hasattr (module.Redis, method ):
newrelic.api.function_trace.wrap_function_trace (module, 'Redis.%s' % method )

[alsotang]

在这里贴代码也是醉了。。。。。我最近发现个网站挺好用的，在此推荐一下： https://gist.github.com/