Introduction

3-way-merge(henceforth 3WM) is popularly used at SCM(Source code Configuration Management) tool. But, there is no standard way of implementation. And most of them is for text(source code). So, there are already lots of study regarding Text-3WM. But, for me, it's very difficult to find study regarding Json-3WM. Someone may think Json-3WM is subset of Text-3WM. And I think it can be one of possible algorithm of Json-3WM. But, is it enough? This article is for this topic.

Assumption

Array vs. Object

At Json, Array is considered as Object. At Javascript, typeof [] gives object. For example, [1, 2] can be represented as {0:1, 1:2}. In this article, Array is handled as one of primitive value, not an object - just like JSON-Merge-Patch(RFC 7386).

Merge

Way of merging two Json object is clearly defined at JSON-Merge-Patch(RFC 7386). Therefore this article is only focusing on 3-way-mergee.

3-Way Merge: Clear

In most common cases, it looks clear. For example,

Mergable

<base>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "g"
    }
}

<our>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "g",
        "h": "i"  # <=
    }
}


<their>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "z"  # <=
    }
}

<merged>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "z",  # <=
        "h": "i"   # <=
    }
}

Conflict

<base>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "g"
    }
}

<our>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "z"  # <=
    }
}


<their>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "y"  # <=
    }
}

3-Way Merge: Not clear

This is main topic of this discussion. Consider following example

<base>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "g"
    }
}

<our>
{
    "a": "b",
    "c": {
        "d": "e",
        "f": "z"  # <=
    }
}

And

<their>
{
    "a": "b"
              # <=
}

It seems conflict. Then, which property is conflicted? /c or /c/f?

Case: /c/f (Incorrect)

In this interpretation, merged result with Accept-Our option (resolve strategy for conflicted property) should be

<merged: accept-our>
{
    "a": "b",
    "c": {
        "f": "z"  # <=
    }
}

What is issue of above? In this case, there is no difference from below version of <their> object.

<their>
{
    "a": "b",
    "c": {}  # <=
}

In case of above version, everything looks clear. Definitely, /c/f is property conflicted. Then, Is it natural that two different changes gives same merged result? I think No. Therefore this is not correct interpretation of given example.

Case: /c

In this case, interpretation of changes are

  • <their>: Deleting /c.
  • <our>: Changing /c/f and /c(sub-property of /c is changed).
    So, /c is conflicted and merged result with Accept-Our option (resolve strategy for conflicted property) should be
    <merged: accept-our>(same with <our>)
    {
      "a": "b",
      "c": {
          "d": "e",
          "f": "z"
      }
    }
    Whole sub-object /c is resolved with /c of <our> object because of Accept-Our merge strategy.

Summary

This article describes private opinion related with 3-way-merge algorithm of Json object. There is NO correct answer. This is just one of suggestion.

블로그 이미지

yhcting

Fabric has great document for it.
See https://hyperledger-fabric.readthedocs.io/en/release-1.2/build_network.html#

But, once you try you own network from scratch, you will face lots of unexpected obstacles.
So, in addition to this guide, I would like to mention some more information. If you are not familiar with docker and docker-compose, you may confuse what is exactly happending inside of byfn(Build Your First Network). Therefore, before staring building your network, please understand what is docker and docker-compose.

One important thing regarding docker-compose is, by default docker-compose uses user-defined bridge network mode. And, instantiation CC means running docker-container. Problem is, container for CC also should join to network in where peer is joined.

Environment variable CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE is for it!

You should NOT forget this!
Without this, container cannot connect to peer nodes!

블로그 이미지

yhcting

Environment: Hyperledger Fabric v1.2

Fabric uses 'npm install --production' to build chaincode docker image.
(https://github.com/hyperledger/fabric/blob/release-1.2/core/chaincode/platforms/node/platform.go#L188)

And run CC by using 'npm start -- --peer.address'
(https://github.com/hyperledger/fabric/blob/release-1.2/core/chaincode/container_runtime.go#L147)

So, only following files are needed to be released

  • javascript files
  • package.json
  • package-lock.json

In case of NodeJs, javascript files should be deployed to peer nodes. So, usually uglifying and optimization is required.
And webpack is most popular tool for this requirements.
And in case that all source codes are bundled to one file - bundle.js - then, releasing only three files are enough!

  • bundle.js
  • package.json
  • package-lock.json

And 'npm start' may look like 'node bundle.js'.

블로그 이미지

yhcting

Hyperledger Fabric v1.2


./users/Admin@example.com/tls/client.key   (*E)
./users/Admin@example.com/tls/ca.crt   (*b)
./users/Admin@example.com/tls/client.crt   (*E)


# AdminMsp {

./users/Admin@example.com/msp/cacerts/ca.example.com-cert.pem   (*a)
./users/Admin@example.com/msp/tlscacerts/tlsca.example.com-cert.pem   (*b)
./users/Admin@example.com/msp/admincerts/Admin@example.com-cert.pem   (*c) (Auth. by (*a))
./users/Admin@example.com/msp/keystore/a331923a189a86dca0832b2841e077a3701cfe0063d6792e88f593a243c3338b_sk   (*C)
./users/Admin@example.com/msp/signcerts/Admin@example.com-cert.pem   (*c)(*C) (Auth. by (*a))

# } // AdminMsp



./orderers/orderer.example.com/tls/server.crt   (*F)
./orderers/orderer.example.com/tls/server.key   (*F)
./orderers/orderer.example.com/tls/ca.crt   (*b)


# OrdererMsp {

./orderers/orderer.example.com/msp/cacerts/ca.example.com-cert.pem   (*a)
./orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem   (*b)
./orderers/orderer.example.com/msp/admincerts/Admin@example.com-cert.pem   (*c) (Auth. by (*a))
./orderers/orderer.example.com/msp/keystore/caec6e5979df95564f57b0a50174cc5004977cbdf3f667d1b9f1e1d5128ff490_sk   (*D)
./orderers/orderer.example.com/msp/signcerts/orderer.example.com-cert.pem   (*D) (Auth. by (*a))

# } // OrdererMsp


./ca/ca.example.com-cert.pem   (*a)(*A)
./ca/74652114cc0ad0b0b9c3365af181f9dc3e240fbe8373ec30f3cd10de4bde6221_sk   (*A)

./tlsca/tlsca.example.com-cert.pem   (*b)(*B)
./tlsca/534fde3483b518f3887590a10553f6b274d23692eee5bf5976f9be4943c3926d_sk   (*B)

./msp/cacerts/ca.example.com-cert.pem   (*a)
./msp/tlscacerts/tlsca.example.com-cert.pem   (*b)
./msp/admincerts/Admin@example.com-cert.pem   (*c)


-------------------------------------------------------

NOTE:
- (*a) (*b) ... are same files.
- (*A) (*B) ... matching crypto (pub/priv matching)


To run operations requiring ‘admin’ policy, ‘AdminMsp’ is required.
That is, cli-node or peer-node having ‘AdminMsp’ can run admin operations.
(CORE_PEER_MSPCONFIGPATH should be path to AdminMsp)
블로그 이미지

yhcting

Note: This review is written while closing very small projects with Hyperledger Fabric(henceforth HLF).


You can easily find articles regarding pros and cons of HLF network architecture - ex. consensus algorithm, permission and so one - comparing with others - ex. Bitcoin, ethereum. So, in this articile, I would like to discuss things in a point of developing Chaincode.

Updating same state at multiple transactions

I think most important and biggest difference in terms of developing Chaincode between HLF and other popular networks, is


In case that several transactions try to update same global state, only one transaction is allowed in a block!


This comes from architectural design of HLF - Orderer(I think this is a kind of debt for hight TPS). Even if time interval of creating block, is very short comaring with other networks, this is very serious constraints. For example, in case of money trading system, only one transfer-transaction is allowed in a block for same account. Because of this characteristics - high TPS with contraints described above, HLF is very good for assets having it's own ID like house, products and so one. But, it's not good for assets requiring couting and calculation like money.

To overcome this constraints, HLF suggests sample codes at their <fabric-sample>/high-troughput. But I think this is not enough. For example, to know current balance, transaction have to read all accumulated variables. And to transfer money, one variable is added to global state. That is, still only one transaction is allowed in a block because read-set to calculate current balance is updated(added)!

Other Misc.

I think followings are not HLF specific issues. But to me, these are also annoying. And I can't find any good way to reduce these pain points at SDKs or libraries for Chaincode.

  • Read/Write operation on global state is very expensive. And reading variable that is updated in same transaction gives it's original value(not updated value)
  • ECDSA is algorithm for digital signature, not for en/decryption.
  • Testing Chaincode requires HLF networks. So, test and debugging cycle takes longer time than expected.

I hope that HLF team provides good solutions for them. And until then, I hope my sample template - hlfcc-node-starter - is helpful to other developers using typscript as their Chaincode-language.

블로그 이미지

yhcting

 ngrx/store 6.0.1 / Angular 6.x


 ngrx/store 를 처음 써 보는 중에 만나 이슈인데... 어찌보면 당연하긴 하겠지만... 뭔가 magic을 기대하기도 했는데.. 어쨌든...


createSelector(<state>, <action>) 이렇게 될때,  <state> 는 하위의 변화를 detecting하지는 못하는 걸로 보인다. 즉.

state: {
    inner: {
        a: true
    }
}

의 경우

createSelect(state => state.inner, inner => inner.a)

이렇게 해 놓고, dispatch를 통해서, state.inner.a = false 로 하면
selector의 Observer가 fire되지 않는다. 즉, inner.a 는 바뀌었으나, state.inner는 바뀐게 아니라는...


이때 states reference 자체가 reducer에서 바뀌는 경우.. 즉

return state;

가 아니라

return {...state}

가 되면 state의 reference다 바뀌므로 Observer가 fire된다.



블로그 이미지

yhcting

Environment: Angular 6.x, Angular Material 6.x


Tooltip애 show 된 이후, Tooltip value에 undefined가 들어가게 되면, 이후 tooltip이 비활성화 되는 것으로 material tooltip component가 처리하는 것으로 보인다.
따라서,
    - html -
    [matTooltip]="myTooltip"

이렇게 한 경우,
tooltip이 'show'된 이후, myTooltip값이
    'a' => 'b' => 'c'
이 순서로 바뀌면, 'c'가 tooltip으로 잘 보이는데,
    'undefined' => 'b'
순서로 replace하면, 'b'가 보이지 않는다. 첫번째 undefined로 인해서, tooltip이 disable되어 버리는 것 같다.

이 경우, mouse를 leave했다가, 다시 enter해서 tooltip이 다시 'show'될 경우, 'b'가 보인다.
즉, 현재 show session에서는 update되지 않고, 다음 'show' session에 보인다.
이것은
    'a' => undefined => 'c'
순서로로 변경할 경우도 마찬가지다 - 당시 show session에서는 'c'가 보이지 않는다.



블로그 이미지

yhcting

Sub module은 내가 어디에 붙어 있는지 몰라야 한다.  즉 module "내부" => "외부" 로 dependency가 없어야 한다. 따라서, Inter-Module Communication은 상위 module을 통하는게 좋아 보인다.


                      +--------------------+
              +------->  Parent  Module    +---+
              |       +---------+----------+   |
              |                 |              |
Parent Module is observing    Notify         Notify
              |                 |              |
    +---------+-+     +---------v-+     +------v---------+
    | Module A  |     | Module B  |     | ...            |
    +-----------+     +-----------+     +----------------+


이 경우, 문제는 Lazy-loading module이다.

Lazy-loading module의 Concept상, Parent Module (이 경우, Root Module)에서 Notify를 할 수 없고 해서도 안된다 - Notify를 하려면, loading을 해야하는데... 그럼 더 이상 lazy loading이 아니다!

(만약 해야할 필요가 있다면, Lazy-loading으로 만들면 안된다. )


따라서, Lazy-loading module의 경우, 일반적인 sub-module의 제약사항에 더불어 추가적으로, 다음의 조건이 필요하지 않나... 싶다.(아직 좀더 많은 공부가 필요하다.)

- App. singleton provider 를 사용하면 안된다.

- module이 loading되었음을 가정한, "외부 => module" 로 communication이 없어야 한다.

<= 이거 상당히 강력한 제약사항이다!!!

그리고, 이건: "Inter-lazy-loading module간 communication이 없어야 한다." 를 의미하기도 한다.


블로그 이미지

yhcting

[ ZIP extended data descriptor and signature issue at Python zipfile library. (Python 2.7.12) ]

Zip format spec을 보면, extended data descriptor쪽이 불명확하게 되어 있다

원칙적이로는 다음과 같은 format을 따른다.(참고: http://www.pkware.com/documents/casestudies/APPNOTE.TXT)

 4.3.9  Data descriptor:

        crc-32                          4 bytes
        compressed size                 4 bytes
        uncompressed size               4 bytes


문제는, De-facto standard로 아래와 같이 정의되는 경우가 대부분이라는 것이다. (http://www.onicos.com/staff/iz/formats/zip.html)

Extended local header:
Offset   Length   Contents
  0      4 bytes  Extended Local file header signature (0x08074b50)
  4      4 bytes  CRC-32
  8      4 bytes  Compressed size
 12      4 bytes  Uncompressed size



[ 7z - 16.02 버젼 ]


CPP/7zip/Archive/Zip/ZipIn.cpp: CInArchive::ReadLocalItemAfterCdItemFull()

    ...
    if (item.HasDescriptor())   // <= 0x8 bit of general purpose flag, is set
    {
      // pkzip's version without descriptor is not supported
      RINOK(Seek(ArcInfo.Base + item.GetDataPosition() + item.PackSize));
      if (ReadUInt32() != NSignature::kDataDescriptor)
        return S_FALSE;
    ...


와 같이 De-facto standard를 따르고 있다.


[ Python zipfile library ]


zipfile.ZipFile 의 write 혹은 writestr 함수를 보면

< writestr 함수 >
    ...
        if zinfo.flag_bits & 0x08:
            # Write CRC and file sizes after the file data
            fmt = '<LQQ' if zip64 else '<LLL'
            self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size,
                  zinfo.file_size))
    ...

와 같이 signature가 빠져 있다. 즉 standard를 따른다!

[ 누구의 잘못인가? ]


AppNote(http://www.pkware.com/documents/casestudies/APPNOTE.TXT)는 아래와 같이 말하고 있다.

      Although not originally assigned a signature, the value
      0x08074b50 has commonly been adopted as a signature value
      for the data descriptor record.  Implementers should be
      aware that ZIP files may be encountered with or without this
      signature marking data descriptors and should account for
      either case when reading ZIP files to ensure compatibility.
      When writing ZIP files, it is recommended to include the
      signature value marking the data descriptor record.  When
      the signature is used, the fields currently defined for
      the data descriptor record will immediately follow the
      signature
     
즉, 원칙적으로는 7z 이 양쪽의 경우를 다 지원하도록 확장되는게 맞는것 같다.
다만 python의 zipfile library역시 recommendation을 따르는게 좋을 것 같으나... 그럴 기미가 안보인다. (Python3.5.2 에서도 zipfile library는 여전히 순수 표준을 따른다.)


[ 언제 문제가 되는가? ]


extended data descriptor를 사용하는 걸로 set된 zip file을 python library로 update할 경우, 혹은 python library로 extended data descriptor를 사용해서 zip을 생성할 경우, 이렇게 생성된 zip file은 다른 popular한 tool에서 사용할 수 없을 수도 있다.
Ex.
    7z 의 경우, 'x'(extract)는 잘 되나, 'u'(update)에서는 'E_NOTIMPL' System ERROR가 발생한다.
    'zip'(pkzip)의 경우 양쪽 모두 잘 지원한다. 다만... 다른 bug가...(>4G => <4G => >4G 버그?)

또한 7z의 경우, 내부적으로 병렬로 pkzip을 수행하는 것으로 보인다.

$ time 7z ...

의 command로 확인해보면, 바로 알 수 있다. 또한 속도도 빠르다!

블로그 이미지

yhcting

Environment : Ubuntu 14.04
Tools : sem, flock(recommend)

> $ cat a.sh
#!/bin/bash

str=0123456789
for i in {1..7}; do
    str=$str$str
done

for i in {1..10}; do
    echo $str >> out                   # (*a)
#    sem "echo $str >> out"            # (*b)
#    flock out -c "echo $str >> out"   # (*c)
done


> $ cat runa
#!/bin/bash
id=$1
for i in {1..10}; do
    ./a.sh&
done


> $ cat runa2
#!/bin/bash

for i in {1..10}; do
    ./runa&
done

> $ ./runa2


check
---------------
$ cat parout | uniq | wc -l
1
---------------
if 1 => OK, otherwise not-syncronized!!

(*a) ==> race condition!! :
(*c) sem ==> error!(sem bug!!)
(*b) flock ==> OK

블로그 이미지

yhcting