[Python] __iter__ method of class at add operator with list...

Language/Python 2017.05.19 20:32


class L(object):
    def __init__(self, lst):
        self.l = lst

    def __iter__(self):
        print('iter..')
        for l in self.l:
            yield l

    def __str__(self):
        return ' '.join(self.l)


l1 = L(['1', '2', '3'])
l2 = []
ll2+= l1
print(' '.join(l2))


$ ./a.py
iter..
1 2 3



Trackback 0 : Comment 0

[Ubuntu/Linux] Python zip library 와 7z 간 교차 사용시 문제

Domain/Linux 2017.03.08 22:08

[ ZIP extended data descriptor and signature issue at Python zipfile library. (Python 2.7.12) ]

Zip format spec을 보면, extended data descriptor쪽이 불명확하게 되어 있다

원칙적이로는 다음과 같은 format을 따른다.(참고: http://www.pkware.com/documents/casestudies/APPNOTE.TXT)

 4.3.9  Data descriptor:

        crc-32                          4 bytes
        compressed size                 4 bytes
        uncompressed size               4 bytes


문제는, De-facto standard로 아래와 같이 정의되는 경우가 대부분이라는 것이다. (http://www.onicos.com/staff/iz/formats/zip.html)

Extended local header:
Offset   Length   Contents
  0      4 bytes  Extended Local file header signature (0x08074b50)
  4      4 bytes  CRC-32
  8      4 bytes  Compressed size
 12      4 bytes  Uncompressed size



[ 7z - 16.02 버젼 ]


CPP/7zip/Archive/Zip/ZipIn.cpp: CInArchive::ReadLocalItemAfterCdItemFull()

    ...
    if (item.HasDescriptor())   // <= 0x8 bit of general purpose flag, is set
    {
      // pkzip's version without descriptor is not supported
      RINOK(Seek(ArcInfo.Base + item.GetDataPosition() + item.PackSize));
      if (ReadUInt32() != NSignature::kDataDescriptor)
        return S_FALSE;
    ...


와 같이 De-facto standard를 따르고 있다.


[ Python zipfile library ]


zipfile.ZipFile 의 write 혹은 writestr 함수를 보면

< writestr 함수 >
    ...
        if zinfo.flag_bits & 0x08:
            # Write CRC and file sizes after the file data
            fmt = '<LQQ' if zip64 else '<LLL'
            self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size,
                  zinfo.file_size))
    ...

와 같이 signature가 빠져 있다. 즉 standard를 따른다!

[ 누구의 잘못인가? ]


AppNote(http://www.pkware.com/documents/casestudies/APPNOTE.TXT)는 아래와 같이 말하고 있다.

      Although not originally assigned a signature, the value
      0x08074b50 has commonly been adopted as a signature value
      for the data descriptor record.  Implementers should be
      aware that ZIP files may be encountered with or without this
      signature marking data descriptors and should account for
      either case when reading ZIP files to ensure compatibility.
      When writing ZIP files, it is recommended to include the
      signature value marking the data descriptor record.  When
      the signature is used, the fields currently defined for
      the data descriptor record will immediately follow the
      signature
     
즉, 원칙적으로는 7z 이 양쪽의 경우를 다 지원하도록 확장되는게 맞는것 같다.
다만 python의 zipfile library역시 recommendation을 따르는게 좋을 것 같으나... 그럴 기미가 안보인다. (Python3.5.2 에서도 zipfile library는 여전히 순수 표준을 따른다.)


[ 언제 문제가 되는가? ]


extended data descriptor를 사용하는 걸로 set된 zip file을 python library로 update할 경우, 혹은 python library로 extended data descriptor를 사용해서 zip을 생성할 경우, 이렇게 생성된 zip file은 다른 popular한 tool에서 사용할 수 없을 수도 있다.
Ex.
    7z 의 경우, 'x'(extract)는 잘 되나, 'u'(update)에서는 'E_NOTIMPL' System ERROR가 발생한다.
    'zip'(pkzip)의 경우 양쪽 모두 잘 지원한다. 다만... 다른 bug가...(>4G => <4G => >4G 버그?)

또한 7z의 경우, 내부적으로 병렬로 pkzip을 수행하는 것으로 보인다.

$ time 7z ...

의 command로 확인해보면, 바로 알 수 있다. 또한 속도도 빠르다!

Trackback 0 : Comment 0

Bash print call stack at ERR(trap)

Language/Bash 2017.03.08 22:03


At head of script file:

(CODE-1)
trap 'on_error "${FUNCNAME[@]} ${BASH_LINENO[@]} $LINENO"' ERR
trap 'on_exit' EXIT

Someone may wonder why don't write code like below because it is easier to handle arguments.

(CODE-2)
trap 'on_error "$LINENO ${FUNCNAME[@]} ${BASH_LINENO[@]}"' ERR
trap 'on_exit' EXIT

I don't know there is any documented information related with this. But, results of my experiments are saying that (CODE-2) doens't work as expected.
In case of (CODE-2), my test shows that only latest function-stack information is passed to 'on_error' trap function.
I don't have any idea about the reason. More investigation is required for this.
But anyway, (CODE-1) works well.
So, you can use those arguments to print function call stack at bash.
You may need to use 'BASH_SOURCE' array too, if your bash uses other files, too.



And there is one interesting case. See follow code.

<< Test environment >>
bash: GNU bash, version 4.3.46(1)-release (x86_64-pc-linux-gnu)
OS: Linux XXXX 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

-------------------------- TEST-1 -----------------------------
<< a.sh >>
function on_err() {
    echo "error"
}

function on_exit() {
    echo "exit"
}

trap on_err ERR
trap on_exit EXIT

myval=$(echo u | grep p)  # global variable


$ bash -eE a.sh
error
exit

-------------------------- TEST-2 -----------------------------

<< a.sh >>
function on_err() {
    echo "error"
}

function on_exit() {
    echo "exit"
}

trap on_err ERR
trap on_exit EXIT

function f0() {
    local myval=$(echo u | grep p)  # local variable in function
}

f0

$ bash -eE a.sh
exit

---------------------------------------------------------------

Even if errtrace is enabled(-E option), 'error' is NOT printed at TEST-2.
Then, is this means 'ERR' is NOT trapped at TEST-2? That is, does 'on_err' not executed?
Let's have a look following code.

-------------------------- TEST-3 -----------------------------

<< a.sh >>
function on_err() {
    echo "error"
}

function on_exit() {
    echo "exit"
}

trap on_err ERR
trap on_exit EXIT

function f0() {
    myval=$(echo u | grep p)  # NOT local variable anymore.
}

f0

$ bash -eE a.sh
error
exit

-------------------------- TEST-4 -----------------------------

<< a.sh >>
function on_err() {
    echo "error" 1>&2  # echo to standard error.
}

function on_exit() {
    echo "exit"
}

trap on_err ERR
trap on_exit EXIT

function f0() {
    local myval=$(echo u | grep p)  # local variable
}

f0

$ bash -eE a.sh
error
exit

-------------------------- TEST-5 -----------------------------


function on_err() {
    echo "error" 1>&2  # echo to stderr
}

function on_exit() {
    echo "exit"
}

trap on_err ERR
trap on_exit EXIT

myval=$(echo u | grep p)  # assign to global variable.


$ bash -eE a.sh
error
error
exit


-------------------------- TEST-6 -----------------------------


function on_err() {
    true  # There is no echo
}

function on_exit() {
    echo "exit"
}

trap on_err ERR
trap on_exit EXIT

myval=$(echo u | grep p)  # asign to global variable


$ bash -eE a.sh
exit

---------------------------------------------------------------

It's very interesting, isn't it?
Further investigation will be continued for this issue.



'Language > Bash' 카테고리의 다른 글

[Bash] 'set -e' inheritance...???  (0) 2017.05.19
Bash print call stack at ERR(trap)  (0) 2017.03.08
Trackback 0 : Comment 0