GAE/Go & Firestore

12 Sep 2020

Daigo Ikeda

Knightso, LLC

Profile

Daigo Ikeda
@hogedigo

Knightso, LLC
http://www.knightso.co.jp/
Shizuoka, JAPAN

2

概要

3

GAE/Go

4

What is GAE?

5

GAE Standard Environment(SE)

6

GAE Second generation(2nd gen)

7

GAE/Go(SE)

FYI:
- GAE Go 1.11 ランタイムが公式には 2nd gen ではなくなった件について

8

Cloud Firestore

9

What is Firestore?

10

旧Datastoreおさらい

手抜きスミマセン🙇

11

Firestore in Datastore mode

12

Firestoreトランザクション

13

Firestoreトランザクションを試してみる

14

Atomic

> Each transaction is guaranteed to be atomic, meaning that transactions are never partially applied. Either all of the operations in the transaction are applied, or none of them are applied.

15

Atomic - OK

16

実行結果

17

Atomic - Rollback

18

実行結果

2020/08/21 04:04:46 わざとエラー
exit status 1
19

500エンティティPut

> Transactions can query or lookup any number of entities. You can create, update, or delete up to 500 entities in each transaction.

20

500エンティティPut - OK

21

実行結果

22

501エンティティPut - NG

for i := 0; i < 501; i++ {
23

実行結果

2020/08/22 14:29:17 rpc error: code = InvalidArgument desc = cannot write more than 500 entities
in a single call
exit status 1
24

グローバルクエリ

> Queries in transactions are no longer required to be ancestor queries.

25

グローバルクエリ- OK

26

実行結果

done. total:500
27

Locks

28

Locks

> Read-write transactions use reader/writer locks to enforce isolation and serializability. When two concurrent read-write transactions read or write the same data, the lock held by one transaction can delay the other transaction.

29

検証コード(各ケース共通)

30

Locks - Get vs Get

31

実行結果

プロセス1

$ ./locks -s 20s Get
2020/09/12 15:20:43 start
2020/09/12 15:20:44 getting...
2020/09/12 15:20:44 got {Value:1 CreatedAt:2020-09-10 14:19:37.872402 +0900 JST}
2020/09/12 15:20:44 sleeping... 20s
2020/09/12 15:21:04 commiting...
2020/09/12 15:21:04 done.

プロセス2

$ ./locks Get
2020/09/12 15:20:47 start
2020/09/12 15:20:48 getting...
2020/09/12 15:20:48 got {Value:1 CreatedAt:2020-09-10 14:19:37.872402 +0900 JST}
2020/09/12 15:20:48 sleeping... 0s
2020/09/12 15:20:48 commiting...
2020/09/12 15:20:48 done.
32

33

Locks - Get vs Put

34

実行結果

プロセス1

$ ./locks -s 20s Get
2020/09/12 16:17:38 start
2020/09/12 16:17:38 getting...
2020/09/12 16:17:38 got {Value:1 CreatedAt:2020-09-10 14:19:37.872402 +0900 JST}
2020/09/12 16:17:38 sleeping... 20s
2020/09/12 16:17:58 commiting...
2020/09/12 16:17:58 done.

プロセス2

$ ./locks Put
2020/09/12 16:17:43 start
2020/09/12 16:17:43 putting...
2020/09/12 16:17:43 sleeping... 0s
2020/09/12 16:17:43 commiting...
2020/09/12 16:17:58 done.
35

ちなみに・・

36

Locks - Put vs Get

37

実行結果

プロセス1

$ ./locks -s 20s Put
2020/09/12 16:26:54 start
2020/09/12 16:26:55 putting...
2020/09/12 16:26:55 sleeping... 20s
2020/09/12 16:27:15 commiting...
2020/09/12 16:27:15 done.

プロセス2

$ ./locks Get
2020/09/12 16:26:59 start
2020/09/12 16:27:00 getting...
2020/09/12 16:27:00 got {Value:1 CreatedAt:2020-09-10 16:24:57.462099 +0900 JST}
2020/09/12 16:27:00 sleeping... 0s
2020/09/12 16:27:00 commiting...
2020/09/12 16:27:00 done.
38

39

Locks - Put vs Get(その2)

40

実行結果

プロセス1

$ ./locks -s 20s Put
2020/09/12 05:09:08 start
2020/09/12 05:09:09 putting...
2020/09/12 05:09:09 sleeping... 20s
2020/09/12 05:09:29 commiting...
2020/09/12 05:09:36 done.

プロセス2

$ ./locks -s 20s Get
2020/09/12 05:09:15 start
2020/09/12 05:09:16 getting...
2020/09/12 05:09:16 got {Value:1 CreatedAt:2020-09-11 05:03:01.615712 +0900 JST}
2020/09/12 05:09:16 sleeping... 20s
2020/09/12 05:09:36 commiting...
2020/09/12 05:09:36 done.
41

42

Locks - Put vs Put

43

実行結果

プロセス1

$ ./locks -s 20s Put
2020/09/12 03:21:30 start
2020/09/12 03:21:31 putting...
2020/09/12 03:21:31 sleeping... 20s
2020/09/12 03:21:51 commiting...
2020/09/12 03:21:51 done.

プロセス2

$ ./locks Put
2020/09/12 03:21:35 start
2020/09/12 03:21:36 putting...
2020/09/12 03:21:36 sleeping... 0s
2020/09/12 03:21:36 commiting...
2020/09/12 03:21:36 done.
44

45

Locks - Get&Put vs Get

46

実行結果

プロセス1

$ ./locks -s 20s GetPut
2020/09/12 04:14:49 start
2020/09/12 04:14:49 getting...
2020/09/12 04:14:49 got {Value:1 CreatedAt:2020-09-11 04:00:30.603741 +0900 JST}
2020/09/12 04:14:49 putting...
2020/09/12 04:14:49 sleeping... 20s
2020/09/12 04:15:09 commiting...
2020/09/12 04:15:10 done.

プロセス2

$ ./locks Get
2020/09/12 04:14:52 start
2020/09/12 04:14:53 getting...
2020/09/12 04:14:53 got {Value:1 CreatedAt:2020-09-11 04:00:30.603741 +0900 JST}
2020/09/12 04:14:53 sleeping... 0s
2020/09/12 04:14:53 commiting...
2020/09/12 04:14:53 done.
47

48

Locks - Get vs Get&Put

49

実行結果

プロセス1

$ ./locks -s 20s Get
2020/09/12 04:18:39 start
2020/09/12 04:18:40 getting...
2020/09/12 04:18:40 got {Value:1 CreatedAt:2020-09-11 04:18:10.5811 +0900 JST}
2020/09/12 04:18:40 sleeping... 20s
2020/09/12 04:19:00 commiting...
2020/09/12 04:19:00 done.

プロセス2

$ ./locks GetPut
2020/09/12 04:18:45 start
2020/09/12 04:18:45 getting...
2020/09/12 04:18:45 got {Value:1 CreatedAt:2020-09-11 04:18:10.5811 +0900 JST}
2020/09/12 04:18:45 putting...
2020/09/12 04:18:45 sleeping... 0s
2020/09/12 04:18:45 commiting...
2020/09/12 04:19:00 done.
50

51

Locks - Get&Put vs Put

52

実行結果

プロセス1

$ ./locks -s 20s GetPut
2020/09/12 04:21:44 start
2020/09/12 04:21:44 getting...
2020/09/12 04:21:44 got {Value:1 CreatedAt:2020-09-11 04:18:45.825498 +0900 JST}
2020/09/12 04:21:44 putting...
2020/09/12 04:21:44 sleeping... 20s
2020/09/12 04:22:04 commiting...
2020/09/12 04:22:05 done.

プロセス2

$ ./locks Put
2020/09/12 04:21:49 start
2020/09/12 04:21:50 putting...
2020/09/12 04:21:50 sleeping... 0s
2020/09/12 04:21:50 commiting...
2020/09/12 04:22:05 done.
53

54

Locks - Put vs Get&Put

55

実行結果

プロセス1

$ ./locks -s 20s Put
2020/09/12 04:32:43 start
2020/09/12 04:32:44 putting...
2020/09/12 04:32:44 sleeping... 20s
2020/09/12 04:33:04 commiting...
2020/09/12 04:33:04 done.

プロセス2

$ ./locks GetPut
2020/09/12 04:32:48 start
2020/09/12 04:32:48 getting...
2020/09/12 04:32:48 got {Value:1 CreatedAt:2020-09-11 04:31:50.387902 +0900 JST}
2020/09/12 04:32:48 putting...
2020/09/12 04:32:48 sleeping... 0s
2020/09/12 04:32:48 commiting...
2020/09/12 04:32:48 done.
56

57

Locks - Put vs Get&Put(その2)

58

実行結果

プロセス1

$ ./locks -s 20s -v 2 Put
2020/09/12 04:41:10 start
2020/09/12 04:41:11 putting...
2020/09/12 04:41:11 sleeping... 20s
2020/09/12 04:41:31 commiting...
2020/09/12 04:41:36 done.

プロセス2

$ ./locks -s 20s -v 3 GetPut
2020/09/12 04:41:16 start
2020/09/12 04:41:16 getting...
2020/09/12 04:41:16 got {Value:1 CreatedAt:2020-09-11 04:41:05.819933 +0900 JST}
2020/09/12 04:41:16 putting...
2020/09/12 04:41:16 sleeping... 20s
2020/09/12 04:41:36 commiting...
2020/09/12 04:41:36 done.
59

ちなみに旧Datastoreだと・・

60

Locks - Get&Put vs Get&Put

61

実行結果

プロセス1

$ ./locks -s 20s GetPut
2020/09/12 09:14:44 start
2020/09/12 09:14:45 getting...
2020/09/12 09:14:45 got {Value:1 CreatedAt:2020-09-11 08:43:20.957081 +0900 JST}
2020/09/12 09:14:45 putting...
2020/09/12 09:14:45 sleeping... 20s
2020/09/12 09:15:05 commiting...
2020/09/12 09:15:05 done.

プロセス2

$ ./locks GetPut
2020/09/12 09:14:50 start
2020/09/12 09:14:50 getting...
2020/09/12 09:14:50 got {Value:1 CreatedAt:2020-09-11 08:43:20.957081 +0900 JST}
2020/09/12 09:14:50 putting...
2020/09/12 09:14:50 sleeping... 0s
2020/09/12 09:14:50 commiting...
2020/09/12 09:15:05 datastore: concurrent transaction
62

ちなみに旧Datastoreだと・・

63

Locks - Mutation: Get、 Update、(エンティティが存在している状態での)Upsert

64

Locks - Mutation: Insert vs Insert

65

実行結果

プロセス1

$ ./locks -s 20s Insert
2020/09/12 09:36:43 start
2020/09/12 09:36:44 inserting...
2020/09/12 09:36:44 sleeping... 20s
2020/09/12 09:37:04 commiting...
2020/09/12 09:37:04 rpc error: code = AlreadyExists desc = entity already exists: app: "b~kni-fs-tx-test"
path <
  Element {
    type: "Sample"
    name: "test"
  }
>
66

プロセス2

$ ./locks Insert
2020/09/12 09:36:48 start
2020/09/12 09:36:49 inserting...
2020/09/12 09:36:49 sleeping... 0s
2020/09/12 09:36:49 commiting...
2020/09/12 09:36:49 done.
67

Locks - Query & Put

68

実行結果

プロセス1

$ ./locks/locks -sa 10s Query
2020/09/12 17:10:47 start
2020/09/12 17:10:47 sleeping... 0s
2020/09/12 17:10:47 querying...
2020/09/12 17:10:48 0: &{Value:1 UpdatedAt:2020-09-11 16:22:50.11876 +0900 JST}
2020/09/12 17:10:48 1: &{Value:2 UpdatedAt:2020-09-11 16:22:59.032873 +0900 JST}
2020/09/12 17:10:48 2: &{Value:3 UpdatedAt:2020-09-11 16:23:08.504272 +0900 JST}
2020/09/12 17:10:48 sleeping... 10s
2020/09/12 17:10:58 committing...
2020/09/12 17:10:58 done.

プロセス2

$ ./locks/locks -k AA Put
2020/09/12 16:55:26 start
2020/09/12 16:55:27 sleeping... 0s
2020/09/12 16:55:27 putting...
2020/09/12 16:55:27 sleeping... 0s
2020/09/12 16:55:27 committing...
2020/09/12 16:55:42 done.
69

70

Locks - まとめ

71

Isolation and consistency

72

Isolation and consistency

> Datastore mode databases enforce serializable isolation. Data read or modified by a transaction cannot be concurrently modified.

73

Isolation and consistency - Put vs Get

74

実行結果

プロセス1

$ ./locks -s 20s -v 2 Put
2020/09/12 14:50:31 start
2020/09/12 14:50:32 putting...
2020/09/12 14:50:32 sleeping... 20s
2020/09/12 14:50:52 commiting...
2020/09/12 14:50:52 done.

プロセス2

$ ./locks Get
2020/09/12 14:50:36 start
2020/09/12 14:50:36 getting...
2020/09/12 14:50:36 got {Value:1 CreatedAt:2020-09-11 14:50:14.517059 +0900 JST}
2020/09/12 14:50:36 sleeping... 0s
2020/09/12 14:50:36 commiting...
2020/09/12 14:50:36 done.
75

76

Isolation and consistency - 遅延Get Vs Put

ドキュメント下記を検証

> Queries and lookups in a transaction see a consistent snapshot of the state of the database. This snapshot is guaranteed to contain the effect of all transactions and writes that completed prior to the beginning of the transaction.

77

実行結果

プロセス1

$ ./locks -sb 10s Get
2020/09/12 15:33:01 start
2020/09/12 15:33:02 sleeping... 10s
2020/09/12 15:33:12 getting...
2020/09/12 15:33:12 got {Value:99 CreatedAt:2020-09-11 15:33:06.947901 +0900 JST}
2020/09/12 15:33:12 sleeping... 0s
2020/09/12 15:33:12 commiting...
2020/09/12 15:33:12 done.

プロセス2

$ ./locks -v 99 Put
2020/09/12 15:33:06 start
2020/09/12 15:33:06 sleeping... 0s
2020/09/12 15:33:06 putting...
2020/09/12 15:33:06 sleeping... 0s
2020/09/12 15:33:06 commiting...
2020/09/12 15:33:07 done.
78

79

Isolation and consistency - 連続 Get vs PutMulti

1. プロセス1がトランザクション開始
2. プロセス2がエンティティA, B, CをPutMulti
3. プロセス1がエンティティAを取得
4. プロセス3がエンティティB, CをPutMulti
5. プロセス1がエンティティBを取得
6. プロセス4がエンティティCをPutMulti
7. プロセス1がエンティティBを取得

80

実行結果

プロセス1

$ ./getabc
2020/09/12 16:22:46 sleeping... 10s
2020/09/12 16:22:56 key:/Sample,A, value:{Value:1 UpdatedAt:2020-09-11 16:22:50.11876 +0900 JST}
2020/09/12 16:22:56 sleeping... 10s
2020/09/12 16:23:06 key:/Sample,B, value:{Value:2 UpdatedAt:2020-09-11 16:22:59.032873 +0900 JST}
2020/09/12 16:23:06 sleeping... 10s
2020/09/12 16:23:16 key:/Sample,C, value:{Value:3 UpdatedAt:2020-09-11 16:23:08.504272 +0900 JST}
done

プロセス2,3,4

$ ./putabc -v 1 A B C
2020/09/12 16:22:50 putting value:1 into entities [A B C]
done
$ ./putabc -v 2 B C
2020/09/12 16:22:59 putting value:2 into entities [B C]
done
$ ./putabc -v 3 C
2020/09/12 16:23:08 putting value:3 into entities [C]
done
81

82

Isolation and consistency - Get&Put&Get

ドキュメント下記を検証

> Unlike with most databases, queries and lookups inside a Datastore mode transaction do not see the results of previous writes inside that transaction.

83

実行結果

$ ./getputget
2020/09/12 04:09:49 got before put: {Value:1 UpdatedAt:2020-09-12 04:09:38.084512 +0900 JST}
2020/09/12 04:09:49 got after put: {Value:1 UpdatedAt:2020-09-12 04:09:38.084512 +0900 JST}
done
% ./getputget
2020/09/12 04:09:57 got before put: {Value:2 UpdatedAt:2020-09-12 04:09:49.166701 +0900 JST}
2020/09/12 04:09:57 got after put: {Value:2 UpdatedAt:2020-09-12 04:09:49.166701 +0900 JST}
done
84

85

Read-only Transaction

86

Read-only Transaction

TODO 🙇🙇🙇

87

トランザクション処理のハマりどころ

88

RunInTransactionは冪等に

例えば・・

89

トランザクション貼り忘れに注意

    key := datastore.NameKey("Sample", "test", nil)

    if _, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
        var entity fstx.SampleModel

        if err := tx.Get(key, &entity); err != nil {
            return err
        }

        entity.Value++
        entity.UpdatedAt = time.Now()

        if _, err := client.Put(ctx, key, &entity); err != nil {
            return err
        }

        return nil
    }, datastore.MaxAttempts(1)); err != nil {
        log.Fatal(err)
    }
90

実行結果

% ./notxput
2020/09/12 04:23:08 start
2020/09/12 04:23:28 rpc error: code = Aborted desc = too much contention on these datastore entities. please try again.
91

トランザクション内での強制終了を極力避ける

    key := datastore.NameKey("Sample", "test", nil)

    if _, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
        var entity fstx.SampleModel

        if err := tx.Get(key, &entity); err != nil {
            return err
        }

        panic("test")
    }, datastore.MaxAttempts(1)); err != nil {
        log.Fatal(err)
    }

> Transactions expire after 270 seconds or if idle for 60 seconds.

92

(おまけ)Transactional Cloud Tasks

93

Cloud Tasks をトランザションに含めたい

94

作戦

95

RunInTransactionラッパー関数

func RunInTransactionWithTasks(ctx context.Context, f func(ctx context.Context, tx *datastore.Transaction) error, opts ...datastore.TransactionOption) error {
    wrapperFunc := func(tx *datastore.Transaction) error {
        txid := id.NewUUIDv4()

        ctx, cancel := context.WithTimeout(ctx, 30 * time.Second)
        defer cancel()

        txStatus := &TxStatus{
            ID:        txid,
            CreatedAt: timeutil.NowJST(),
        }

        if err := c.MutateInTx(tx, datastore.NewInsert(Key(txStatus), txStatus)); err != nil {
            return xerrors.Errorf(": %w", err)
        }
(つづく)
96

(つづき)
        ctx = WithStatus(ctx, txStatus)
        return f(ctx, tx)
    }

    if _, err := client.RunInTransaction(ctx, wrapperFunc, opts...); err != nil {
        return xerrors.Errorf(": %w", err)
    }

    return nil
}
97

Task Add

headers := make(map[string]string)

status := libtx.Status(ctx)
if status != nil {
    headers[TaskTxIDHeaderName] = status.ID
    headers[TaskTxDispatchTimeHeaderName] = status.CreatedAt.Format(time.RFC3339Nano)
}

task := &taskspb.Task{
    MessageType: &taskspb.Task_AppEngineHttpRequest{
        AppEngineHttpRequest: &taskspb.AppEngineHttpRequest{
            HttpMethod:  taskspb.HttpMethod_POST,
            RelativeUri: relativeUri,
            Body:        body,
            Headers:     headers,
            AppEngineRouting: &taskspb.AppEngineRouting{
                Service: service,
                Version: version,
            },
        },
    },
}
(つづく)
98

(つづき)
req := &taskspb.CreateTaskRequest{
    Parent: queuePath,
    Task:   task,
}

if _, err := client.CreateTask(ctx, req); err != nil {
    return xerrors.Errorf(": %w", err)
}
99

Tasksハンドラ

txID := r.Header.Get(tasksutil.TaskTxIDHeaderName)

if txID != "" {
    dispatchTime, err := time.Parse(time.RFC3339Nano, r.Header.Get(tasksutil.TaskTxDispatchTimeHeaderName))
    if err != nil {
        w.WriteHeader(http.StatusBadRequest)
        return
    }

    txStatus := &libtx.TxStatus{ID: txID}
    if err := client.Get(ctx, txStatus); xerrors.Is(err, domerr.ErrNotFound) {
        if time.Now().Sub(dispatchTime) > tx.TxTimeout*2 { // トランザクションタイムアウトの2倍超えたらタイムアウト
            return
        } else {
            w.WriteHeader(http.StatusLocked)
            return
        }
    } else if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
}
100

Thank you

12 Sep 2020

Daigo Ikeda

Knightso, LLC

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)